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but in the concept of similarity that applies to all content in the same manner. That 
way, homophily 


serves as an alibi for the inequality it maps, while also obviating politics: homophi- 
ly (often allegedly of those discriminated against)not racism, sexism, and inequa- 
litybecomes the source of inequality, making injustice ‘natural’ and ‘ecological?’ 
(Chun 2018: 76) 


Algorithmic filtering, which is an exemplary case of what Rouvroy calls the “re- 
configuration of informational [...] architectures” (Rouvroy 2013: 155), is criticized 
regarding a universal trait of human subject formation. Chun shows that it is nec- 
essary to take the social situation of subjects, which enable racism, sexism, in- 
equality into account. Another case in point would be the infamous analysis by 
ProPublica, which has shown that a recidivism prediction software was biased 
against blacks (Angwin and Larson 2016). This case has been discussed almost too 
much, so I just want to highlight that the software did not use any racial features 
as input. Thus, even if the efficacy of algorithms does not work in terms of race, it 
still addresses and produces race. 

In order to overcome the line of critique mentioned in the beginning, which 
implies a neutral technology spoilt by biased data, it is necessary to show how any 
kind of media and AI in particular engage with socially and culturally situated 
subjects—including race. 


4. Situated subjects 


In her book on what she calls “habitual new media,” Chun describes data analytics 
and their turn away from individuals quite similar to Rouvroy. Her analysis cen- 
ters on the concept of habit: rather than focusing on an individual subject, data 
analytics try to grasp habits, established ways of acting, and consequently tries 
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to form and influence these habits. In order to achieve this, they focus on the cor- 
relation between habits rather than individual acts or even individual patterns. 
“Through this, individual actions become indications of collective patterns rath- 
er than exceptions.” (Chun 2016: 57) These patterns are the object of optimization, 
quite similar to Rouvroy’s description of the reconfiguration of architectures and 
environments in order to achieve certain behaviors. 

Here, I cannot follow the detailed conceptual work in which Chun engages 
with the notion of habit. However, I want to follow her suggestion to connect this 
take on habit from media theory with thoughts on habit that relate to alterity: 


habit is publicity: it is the experience, the scar, of others that linger in the self. Ha- 
bits are remnants of the past—past goals/selves, past experiences—that live on in 

our reactions to the environment today, as we anticipate tomorrow. Through habit 

we inhabit and are inhabited by alterity. (Ibid.: 95) 


Chun encourages us to ask how such habits are changed through recent develop- 
ments in digital media and how they can change again in order to change society 
(ibid.: 8). This implies that not habit per se is the problem, but differences among 
habits. However, Chun herself does not take these differences serious enough. 
Her main preoccupation are liberal injunctions to protect the subject from alterity 
and technological impositions. By fusing both, she urges to find ways to “inhabit” 
our habitualized relations to others, which includes to “warily embrace” the many 
new flows of data, connections, configurations of subjectivity. Here she has a 
much more positive outlook on technology than Rouvroy. Yet, she underestimates 
how any form of exchange and ensuing subjectivity is formed by power—not just 
the private, liberal space. Some socio-technical positions are quite hurtful to in- 
habit. Thus, in the following I want to suggest a middle ground, which however 
shares the outlook that changes in the ways we perceive and the ways we (can) act 
in a given situation are not only the aim of algorithmic means of governing. They 
are a fundamental way how subjectivity works. This is analyzed in detail by Linda 
Martin Alcoff in her book visible identities. 

Alcoff starts from the Foucauldian insight that power is not just an imposition 
from the outside. Rather, being a thinking and acting subject also means to be 
situated in power. However, contrary to Foucauldian analysis which focuses on 
the disciplinary subjection under norms, Alcoff shows via a theory of alterity and 
habitualization that our perceptions and actions are formed by the practices we 
perform and by the situations we have found ourselves in. Our past experiences 
leave traces that Alcoff describes in line with central insights from what is com- 
monly discussed as theory of social practices (Reckwitz 2002): “[T]he interpretive 
horizon that constitutes our identity is undoubtedly constituted [...] by a wealth of 
tacit knowledge located in the body.” (Alcoff 2006: 106) Such tacit knowledge and 
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habitualizations have their location in practices. They are not necessarily imposed 
on us, rather they are the growing residue of the way we act—or are forced to 

act. The latter of course remains important, but is not the only way how habitu- 
alization comes about. It is an integral part of the way we make meaning of our 

situation and how we structure our actions. A lot of these ways of perceiving and 
acting come from others—via education, the various contexts we live, work, play, 
learn, etc. All of these contexts or situations are structured by collective practic- 
es. Practices in which we do something but at the same time attain a subjectivity. 
Others tell us—more or less implicitly—who we are, what we become or should 

become by doing certain things, what is apt or usual for “someone like you” etc. As 

Alcoff states: 


Part of what the collective praxis creates are aspects of the self. Our preferences, 
our dispositions toward certain kinds of feelings in certain kinds of situations, 
what typically causes fear, anxiety, calmness, anger, and so on, are affected by 
our cultural and historical location. Sometimes people take such internal feelings 
as proof of a natural origin, as when a homosexual kiss elicits feelings of disgust. 
The feelings may well be quite real, but this is not proof that homosexuality is un- 
natural; physical reactions can be altered by knowledge and acquaintance. This 
example suggests the most powerful role that the other plays in self-formation: 
the character of the other determines in no small part the self. (Ibid.: 115) 


Regarding theories of the subject, it is often important to highlight this influence 
against ideas of innate characteristics or the demand to become as self-reflexive 
as possible. Then it suffices that “the situation” of the subject is important—but 
not so much what that situation actually is. Alcoff highlights that the practices we 
become habituated in are structured by all kinds of social difference. She mainly 
analyses race and gender, but points at social strata, education and financial re- 
sources as others. Thus, apparently quotidian practices are different for subjects 
inhabiting different social positions. E.g. she lists all kinds of things that are par- 
ticular for women, with regard to the work of Simone de Beauvoir and Iris Marion 
Young: 


There is not only throwing and sitting, but standing, walking, running, patterns of 
conversation involving interruptions and dominating the topics, perceptual orien- 
tations that can encompass sideline issues so as to notice household dirt, distres- 
sed children, bored interlocutors, and so forth, as well as the very interior expe- 
rience of one’s own emotional subjectivity. (Ibid.: 106) 


She has similar lists for race and cross cultural and intersectional indices (ibid.: 
106 et. seqq.). Alcoff describes that we perceive situations, spaces and persons dif- 
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ferently, depending on our preceding experiences, the cultures and meanings in 
which we have moved. We enter a subway differently as man or a woman, as per- 
son with white skin or person of color. Here, cultures and meanings should not be 
understood as externalizable structures. They only persist in collective practices 
and particularly in what Alcoff calls “perceptual practice” (ibid.: 115). 

It is of course possible to reflect and to engage with one’s own habitualiza- 
tion and the practices in which this happens—but not by rendering them fully 
transparent to oneself. We can act very consciously of the fact that our perception 
and the possible forms of action are deeply intertwined with contingent practic- 
es. Nevertheless, these practices are the very context in which meaning and per- 
ception emerge. Furthermore, experiencing something means to be somewhere 
and thus does not only enable knowledge, but also the possibility to be changed 
in one’s subjectivity: “Knowing is a kind of immanent engagement, in which one’s 
own self is engaged by the world [...] rather than standing apart and above.”(Ibid.: 
111) Thus, when we attempt to engage with our own situation, practices form both 
the context and the site of this engagement. In consequence, habitualization can 
only yield to another form of habitualization: 


The phenomenal world constantly folds back on itself, adding to what has come 
before and what remains still in the background of the present moment; the pastis 
that which has been surpassed, yet remains within. There are no complete breaks 
or total separations, only folds within a continuous cloth, pregnant with latent 
meaning. (Ibid.: 110) 


This also entails that a lasting change of subjectivities cannot be based on individ- 
ual attempts. Rather, the practices, the ensuing social relations need to be changed 
in order to bring about different forms of habitualization and subjectivities: 


Experiences matter, but their meaning for us is both ambiguous and dynamic. We 
are embodied, yet not reduced to physical determinations imagined as existing 
outside of our place in culture and history. This account helps to capture the dialec- 
tics of social identities, in which we are both interpellated into existing categories 
as well as making them our own. (Ibid.: 111) 


This analysis of situatedness, also the situatedness of social change has conse- 
quences for the kind of politics that Rouvroy advocates. Alcoff denies the neces- 
sity for an account of (human) beings as always in excess, or a “pure capacity of 
negation or of flight” (ibid.: 112). Even if such ideas of politics are deeply inspired 
by critiques of the subject, Alcoff contends that they still contain remnants of the 
“dualism” that inspires liberal accounts, which try to somehow separate the indi- 
vidual from others or society. However, the habitual situatedness within practices 
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is not just part of oppressive and determining identities—although these are in 
the foreground of Alcoff’s discussion. They are part of any subjectivity, includ- 
ing those with which we identify, in which we find pleasure, friendship, solidarity, 
luck. In consequence, to attain these we do not need to exceed situatedness, we 
just need to change the situation. In Alcoff’s words: “Moral agency, subjectivi- 
ty, and reasoning capacities are made possible within social networks of certain 
types. There is no amorphous substance or pure capacity lying pristine below the 
layer at which social constructions of identity take hold.” (Ibid.) 


5. Situated Subjects, Al and Politics 


Alcoff herself does not discuss media and technologies. However, her thinking 
is deeply inspired by Merleau-Ponty’s phenomenology, which contains the medi- 
atized structure of experience at its core—represented by the infamous example 
of a blind person’s stick (Merleau-Ponty 1962: 152, see also Alcoff 2006: 188). Thus, 
Alcoff’s thought can be easily amended with the necessary reflections on media 
technology. 

In his discussion of interfaces, Christoph Ernst shows that interaction with 
digital technology via interfaces implies a situated subject, including the body 
(Ernst 2017: 100). Interfaces only work because they can address implicit knowl- 
edge which is rooted in practices and thus is structured by social rules (ibid.: 102). 
Interface research and design even tries to consciously address that using what 
Ernst calls in reference to cognitive science a “conceptual model” (ibid.). While this 
bears the potential of manipulative attempts, it is not manipulation per se but a 
necessity for an interface to work, i.e. to do justice to the fact that interfaces do not 
just interact with generic human beings but concretely situated subjects. 

Ernst discusses interfaces, not the more abstract adjustment of architectures 
or environments that Rouvroy emphasizes, which work through “stimuli and sig- 
nals that produce reflex responses”. However, if this efficacy is precisely the de- 
fining factor of technologies in algorithmic governance, they need to connect to 
the habitualized subjectivities not unlike interfaces (see also Distelmeyer 2017). 
Thus, even if these technologies do not aim at a set of norms and ensuing subjec- 
tivity, they are still entangled with situated subjects. 

This also is confirmed by Chun’s observation that predictive analytics is tied to 
habitual practices, which I have cited above. Using Alcoff’s theory, we now return 
to the point that habitualization itself is not the problem. That a lot happens on a 
pre-conscious and habitual level, does not mean that the applications of AI work 
deterministically on us. Rather they interact with structures of perception and 
action that can certainly be influenced by algorithms, but that are also character- 
ized by a pre-formed depth that results from prior experiences. This can change 
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the previsioned result of algorithmic governance in many ways, ensuing in fric- 
tionless, almost unnoticed influence, as well potentials to inhabit and embrace 
and potentially evolve one’s situation as Chun suggests—but also many kinds of 
tensions, hurt and resistances. This is the main point here. Subjects are concretely 
situated subjects and algorithmic governance, particularly because it acts one su- 
pra-subjective level connects quite differently to the various forms of subjectivity. 

This already starts on the level of perception: For example, EU citizens that are 
not recognized by automated border control terminals that use AI based face rec- 
ognition will immediately see this as malfunction of technology. Migrants might 
perceive this as a threatening decision. Also, the even less tangible adaptations of 
environments connect to situated subjects. This is precisely the reason why appli- 
cations of Al are not neutral. Not just because they are based on biased training 
data; but because they connect differently to different forms of subjectivity. The 
algorithmic filtering of news is problematic because it connects better to certain 
subjects and communities structured by hate and othering than to other forms. 
Recidivism prediction enlarges and continues a security system that is based on 
race discrimination. John Chenney-Lippold has shown that the algorithmic selec- 
tion of merchandise based on a machine learning system that tries to predict the 
users gender connects better to heterosexual, commodified forms of gender than 
others (Cheney-Lippold 2011). 

Al has yielded many technologies that have enhanced the efficacy of technolo- 
gies in the sense that they directly impact the way we perceive and act in the world. 
This impact, however, does not amount to a loss of subjectivity in general. Rather 
it reconfigures different forms of subjectivity in different manners. The norma- 
tive source of critique then does not lie in a difference between a new form of 
subjectivity under algorithmic governance and one that is somehow beyond that. 
Rather, the source of critique lies in the differences that already exist between 
subject positions, and the many ways in which they are shifted through technol- 
ogy. Chun’s suggestion to “warily embrace” this situation can be one way of try- 
ing to achieve a change for the better in such a situated manner. However, other 
ways lie in the refusal to accept a situated subject position, which might include 
demands for privacy protection as well as ceasing to use particular technologies 
altogether. These demands will need a socio-technical index. That is, they are not 
the demand to return to an independent subject position like the liberal strands 
that both Chun and Rouvroy criticize would have it. Still, privacy, cloaking of data, 
refusing to be implicated in automated analysis might be a necessary resource to 
find better and viable situations for persons whose subject position becomes en- 
tangled with applications of Al in hurtful, abusive, disempowering ways. 

To repeat, the challenge of critique is not to escape situatedness but to change 
the situation. Alcoff’s reflection shows that this will always be a situatedness with 
others; and as my amendment of her theoretical outlook illustrates, it will always 
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bea situatedness with technology. Thus, in the end this amounts to a political and 
democratic challenge. Our situation is always already related to others. Applica- 
tions of AI make that very clear: they focus on relational data and as data driven 
technologies only make sense at the aggregate level. 

At the same time, as Alcoff shows, is it impossible to fully reflect that situat- 
edness and relationality. It is not a system or environment but an encroachment 
of many different “past goals/past selves—past experiences” as Chun writes. This 
creates many differences in perception and possibilities for actions for each sub- 
ject. Thus, issues of epistemology and of power are fused. In this sense, the polit- 
ical challenges are first to get to get to know the situation of others, the way that 
technologies connect to their subject position. In a second step these positions 
need to be reconciled to achieve a new and better configuration of technology. This 
needs to be a democratic solution, not in the sense of finding a compromise be- 
tween pre-existing interests, but in the sense that subjects always already form a 
related, socio-technically situated plurality. 
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Introduction 


Frank Pasquale’s 2016 book The Black Box Society is now considered a landmark 
study in law-related disciplines, in the social sciences and beyond. The topic in 
itself, digitalization and the invasiveness of the internet, is of the utmost impor- 
tance. The book reveals the inversion of operational secrecy by digital platforms 
and the extensive access to users’ private data. Facebook, Google and the like 
collect and aggregate bits and bytes of information to create massive profiling 
schemes, the modus operandi of which little is known. Problematically, as private 
actors, they acquire massive data and “knowledge” about the society and our in- 
dividual behaviors that we don’t. To that complex and most certainly unpleasant 
reality, Pasquale’s analytical rigor and finesse contribute valuable insights on po- 
tential regulations and on the possibility of developing a smarter citizenry. How- 
ever, we want to argue that there is another, broader and maybe more cultural 
reason why The Black Box Society draw so much attention—thus further explaining 
the success of the book. The image of a concealed, networked entity is evocative of 
some common fears. It captures a sense of “loss of control” vis-a-vis the latest au- 
tomation processes. Such an algorithmic black box, in other words, taps into a dif- 
fused anxiety regarding what is to be called a “known-unknown”, i.e. something 
we recognize to be a mostly hidden form of knowledge production. The image of a 
black box is a disenchanted one; here lies its strength as well as its weakness. 
While mostly in line with Pasquale’s effort to decipher the opaqueness of our 
data-driven world, it also appears significant to question the limits of the black 
box as a heuristic if not holistic image. Scholars such as Geiger (2017), Sudmann 
(2018), Burrell (2016) or Bucher (2016) have explored this territory. The latter, for 
instance, has argued that “the widespread notion of algorithms as black boxes 
may prevent research more than encouraging it”, noting that the notion is “too 
readily used” (84). She then calls for critical scrutiny of algorithms and algorith- 
mic systems using a three-step method: i) “do not fear the black box”; ii) “do not 
expect the solution to be inside” and iii) “consider the boxing of the box”. Whereas 
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the first step could be understood as encompassing the entire process, the others 
could be conceived as forming a complementary pair, together examining the in- 
side and outside of the box. Moreover, such an approach is particularly suited to 
analyzing the recent shift towards deep learning algorithms, but also to machine 
learning techniques of different sorts, and everything nowadays labelled artificial 
intelligence. How and to what extent these algorithms are practically and symbol- 
ically different from the previous so-called ‘generations’. What would they entail 
in terms of opacity, ambiguity and vagueness? Where, what, and whom should we 
look at to develop critical insight and robust interpretation? These questions, once 
docked to Bucher’s steps, could serve as guide to this chapter. 

Examined closely, the idea of “not expecting the solution to be inside” match- 
es perfectly with the ingrained logic and historical development of deep learning 
algorithms. “Open sesame!” is a task that cannot be programmed, or “learned” 
for that matter. Despite its biological inspiration and the romantic-teleolog- 
ical accounts of the field’s historical development (Rosenblatt, 1958; Hinton & 
al. 2006), the fact remains that what stands for “learning” is in fact adaptation and 
self-tweaking. The mathematical structure modifies itself while interacting and 
coping with the data stream coming from the outside world (Litvinski 2018; Lheu- 
reux & al. 2017). Backpropagation, recursive loops and other subtleties thus not 
only reflect but also enact a reality in flux. Another way of looking at such uncer- 
tainty is with the discrepancy between the more classical symbolic approach to AI 
and today’s connectionist or neo-connectionist shift (Cardon & al. 2018). Whereas 
the first relied on deduction, explicit modeling, abstract rules and programmable 
languages to create a logical and formal mode of reasoning, the second is based 
on induction, whereby connected hypotheses and approximations produce “opti- 
mized” perceptions and predictions about what is going on in the data, inasmuch 
as data translates into improved rates of predictability (Mackenzie 2017, Sudmann 
2018). Layers of non-linear calculus thus inform something of a “deep” but shallow 
architecture which does not necessarily form an inexplicable AI, but which pushes 
the limits of its explicability further away. If not fully black, the box of current 
Al is very grey, to say the least. This can also be seen in the problems scholars are 
now facing concerning the reproducibility of small-scale theories, where current 
practices of publication generally prevent them from sharing both source code 
and training database, or to address the hazards related to the arbitrary setting 
of hyperparameters, or even the unavoidable randomness inherent in the process 
of generating training values (Hutson 2018). Managing and massaging that much 
data is never an easy task, especially not in an experimental environment, and 
even less in the real world where the saying “garbage in, garbage out” remains 
thoroughly valid. The multiple problems nowadays with bias fall under the same 
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category, and could serve as one last example here, namely that the box cannot by 
its very definition be the solution to something bigger than itself.’ 

The internal problems sketched above might not even compare to what is at 
stake with Bucher’s insight probing to “consider the boxing of the box”. In fact, 
there is a long history of social sciences in general and in STS in particular to de- 
vote a great deal of attention to everything surrounding a given piece of technol- 
ogy (Bijker & al. 2012). In the specific case of deep learning algorithms, it is all the 
more important to remember that “they are embedded in larger, far more complex 
assemblages” that never cease to influence their shape and content (Gillespie 2014: 
3). The question, then, is how to make sense of such molding pressures and the 
kind of opacity they produce. It is about the complexity of a given context or a 
given “ecology”, yet we want to argue that the best way to consider such boxing is 
through a networked approach. As stated elsewhere, “[...] there is not one box, but 
multiple boxes. The opacity of algorithms is more precisely expressed in different 
forms of opacity, all of which, in specific ways, are contingent on the in-betweenness 
of a plethora of actors, both human and non-human” (Roberge & Seyfert 2018: 2; 
Latour 1987). First, it is difficult not to acknowledge an intense division of labor 
within this domain of innovation—a situation that often translates into develop- 
ers working on a dataset without fully knowing for whom, to which end and why. 
Here agency is divided amongst many little hands. Second, a networked approach 
would consider the actual implementation of deep learning tools and techniques 
as more in flock than in a row, adjusting to one another more than collaborat- 
ing. This has been well documented in the literature on algorithmic finance, for 
instance, where competing stakeholders deploy “algotrading” tools to bolster at- 
tack or defense maneuvers (Seyfert 2018; Knorr-Cetina & Preda 2011; Castelle & al. 
2016). Whether social sciences will be more attentive in the future to the combined, 
butterfly-like effects of all these actors’ efforts into “boxing the box” remains to 
be seen. However, considering the understudied state of this phenomenon we 
urgently need to give more attention to the management, ordering and decision 
processes that shape what deep learning algorithms come to be about in the real 
world. More straightforwardly, the political economy of AI is one of our biggest 
and most opaque boxes today. In this chapter, we intend to contribute to the on- 
going debate by analyzing what is at stake in this new form of socio-technical gov- 
ernmentality, i.e. what are the tensions, struggles, efforts at coopting knowledge, 
power, etc. Taking the Montreal AI hub as a case study, and following a 2016-2018 


1 Of late, IBM has announced that it would allow access to a library comprising over two million 
images for facial recognition training with the hope that enhanced accuracy would help curb bias. 
The position of NGOs such as the American Civil Liberty Union in that case and in other similar ones 
is that better facial recognition is still bad news for minorities facing discrimination across a vari- 
ety of social settings. See Browne 2019. 
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ethnographical investigation’, we will focus on how stakeholders deploy multiple 
strategies and resources, including the building of legitimacy through symboli- 
cally-laden media operations. Our is thus empirical, while also network approach 
being theoretically informed; in that sense we hope to answer calls by preeminent 
scholars to develop critical thinking through studies which are in situ (Kitchin 
2014; Mackenzie 2018). 


I. Governmentality—What about it, and what Does it Change 
to the Study of Deep Learning? 


Debates surrounding the increasing complexification of today’s political economy 
and how it should translate into new understandings of power gain a great deal 
of intelligibility once one adheres to Michel Foucault’s concept of governmentali- 
ty (2004a, 2004b). This practice-oriented analysis of the ‘problem of government’ 
has allowed scholars to re-orient their focus on “the ceaseless transactions which, 
variably, modify, displace, upset, or insidiously shift the funding, the investment 
modalities, the centers of decision, the forms and types of control, the relations 
between local and central authorities, etc.” (Foucault, quoted in Lascoumes 2004: 
3; our translation.) Indeed, the French philosopher has had a pivotal role in iden- 
tifying—at least—these three logics: i) how power and knowledge are inseparable, 
ii) how these introduce mobile and networked dynamics, and iii) how all of this 
allows to think about authority as enacted by and as a set of technologies. That 
said, the difficulty with Foucault is that he never properly wrote about the digital. 
Of late, it is Mackenzie who has endeavored to apply the concept to the study of 
what he calls ‘machine learners’, i.e. naive Bayes classifiers, decision trees, neu- 
ral networks, and a range of others that fall under the broad category of AI (2013; 
2017; 2018). All of this, according to him, corresponds to a “data practice that re- 
configures local centers of power and knowledge by redrawing human-machine 
relations” (2017: 9). How research, development and implementation is organized; 
by whom, for what purposes, and through which means and discourses, is dif- 
ferent from London to the Silicon Valley, or China and Canada for that matter. 
Likewise, how power relations and the distribution of authority are shaped spe- 
cifically by the structure of its organizations and institutions varies from subfield 
to subfield—finance, military, transport, etc. Mackenzie is thus very helpful by 
providing such ecosystemic, if not ecological views. At the same time, in his book, 
he runs the risk of over-emphasizing an internal examination of the technology, 


2 We totalized 12 interviews with machine learning specialists, 4 additional ones with scientific 
journalists, and over 400 articles from local francophone and anglophone newspapers and 
monthly publications. 
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and thus is only partially able to ‘consider the boxing of the box’—to refer to Bu- 
cher once again. 

What would it take to be able to provide insights that would be both local and 
‘architectural’—that is, able to demonstrate how particular constructions and 
transformations occur from the outside in? One such way is by looking at the dis- 
tance between governmentality and what is deemed today as ‘governance’, and 
how, in fact, the latter is the topic of the former. In Montreal and probably else- 
where, the discourse related to governance enjoys great momentum, as the idea 
itself serves as a sort of empty-signifier that can tactically be given meaning. Gov- 
ernance, like progress or innovation, readily means “good governance” and many 
stakeholders involved in the construction of the Montreal hub conflate the two 
in order to bolster the institutional-public support for market-oriented develop- 
ments in deep learning, the details of which will be presented shortly. For now, 
suffice to say that the very idea of a deep learning governance in Quebec’s metrop- 
olis seeks to implicate pretty much everyone as “partners” in a game of collective 
self-management and purposive social change. In play is what scholars such as 
Walters have identified as “[an] emphasis on self-governing networks” drawing 
heavily on “the imagery of cybernetics and complexity theory” (2004: 29-30; see 
also Simard 1979 for a similar theoretical approach applied to Québec). Power and 
authority here are conceived as enablers: they allow for the circulation of resourc- 
es, not for their constraint or restriction. As will be made clear below, everything 
related to ethics—the industry-backed Partnership on AI or the Montreal Decla- 
ration for a Responsible Development of Al—is tainted by an idea of self-regula- 
tion and its distinctive way of translating into a loose, sickly effort to not legislate. 
Power and politics have not disappeared for that matter; while governance might 
present itself in the best light, as lightweight government at a distance, the point is 
that it represents itself as an efficient, if understudied form of governmentality. 

What is it about the Montreal deep learning hub that makes it worthy of sci- 
entific analysis? Part of the answer relates to the fact that Quebec is a rather small 
society, well developed but still marked by the concentration of its elites—so- 
cial, political, economic, cultural, etc. As for the historical context in which the 
province has addressed the most recent “AI awakening”, it is important to recall 
the role of Canada’s CIFAR in subsidizing deep learning research, even when the 
technique was highly unfashionable (Hernandez 2014; Cardon & al. 2018; Enge- 
mann and Sudmann 2018). Star scientists such as Hinton (University of Toronto), 
former students Bengio (Université de Montréal) and to a lesser extent?, LeCun 
(NYU and now Facebook) are both the inheritors and the best promoters of what 
is now a C$125 million pan-Canadian AI strategy. When, for instance, the talk of 


3 LeCun has worked less in Canada and more in France and USA in recent years, although he still 
enjoys important media coverage. 
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an ongoing “AI revolution” emerged in Quebec’s francophone mediascape, Bengio 
himself came to be introduced as one of the leaders of Montreal’s AI ecosystem, 
itself presented as one of the most world-renowned hubs of cutting-edge AI inno- 
vation (See Bourgault 2017). This peculiar dynamic can be further demonstrated 
by the fact that its name appears in 126 of 161 articles focusing on AI developments 
published by Montreal newspaper La Presse between May of 2016 and July of 2017 
(Bourgault 2017). The point here is that, when considering the Quebec’s AI field, 
two correlations appear clearly: firstly, between the emergent rhetoric of a revolu- 
tion and the rise of a charismatic leader; and secondly, between the accentuated 
hype surrounding deep learning and AI and the capacity of local actors to rapid- 
ly set in motion the relevant institutions. “Hype is low on informative content,” 
scholar Guice rightly observes, “but directly states the relevance of the informa- 
tion to a social context” (1999: 85). In order to bolster the Montreal hub, former 
Quebec’s Economy and Innovation Minister Anglade noted that her government 
would not “sprinkle” public investment (Rettino-Parazelli 2017). That led first to 
the creation of an advisory committee and, subsequently, of an AI Cluster initially 
equipped with a budget of C$100 million. The two most interesting facts about 
the cluster is that it devoted 80% of its funds to Bengio-directed, Université de 
Montréal-led MILA (Montreal Institute for Learning Algorithms) all while being 
officiated by Breton—Université de Montréal’s dean—and well-known business- 
man Boivin, who several months later also became the head of MILA’s board (see 
IA.Quebec 2018). This suggests that in this particular context, and in this rath- 
er short period of time, what good governance meant was delivering efficiency; 
whereas a broader, more reflexive and critical perspective would instead have 
interrogated what it means in terms of circulating elites, and why the effort to 
maximize efficiency still needs to justify and legitimate itself through at least the 
appearance of duly-conducted administrative processes. 

Another way of considering ‘the boxing of the box’ in the Montreal case is to 
have a look at the conjunction between efforts geared towards the launch of the 
aforementioned Montreal Declaration for a Responsible Development of AI and 
the creation, in late 2018, of the International Observatory on the Societal Impacts 
of AI. Fully endorsed by the government and its main scientific institutions, both 
make claims to an epistemological posture of “knowledge co-construction” with 
the public, the different stake-holders, etc., that in practice serves as a malleable, 
if not shallow, signifier. The Declaration, for instance, proposes a list of ten princi- 
ples that are all more general and abstract than the other, with some overly naive 
or in contradiction with the current economic reality of deep learning—Principle 
6.2 for instance states that “AI development must help eliminate relationships of 
domination between groups and people based on differences of power, wealth, 
or knowledge” (IA responsible 2017). For its part, the Observatory is still nascent, 
but in its very constitution already signals a poor understanding of social sci- 
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ences’ role in studying social impacts, with for instance more members coming 
from computer sciences than sociology and communication studies altogether. 
Importantly, what the Observatory and the Declaration have in common is the 
cybernetic view of governance introduced above. On the one hand, the manage- 
ment of knowledge production obliterates any notion of checks and balances or 
arms-length regulatory principles, notions central to the very idea of modernity. 
On the other hand, it appears that all current virtue signaling efforts, including 
the Declaration, the Partnership for AI and the like, emerge as what Wagner calls 
“an escape from regulation” (2018).* All in all, the Quebec government’s involve- 
ment in the development of its Montreal hub is one not of creating barriers and 
obstacles, but rather one to usher and foster the circulation of whatever is deemed 
‘positive’, namely any twists and turns that exhibit a form of action from the gov- 
ernment or the stakeholders, knowing that the legitimacy of who gives reflects on 
who receives and vice versa. 


Il. The ‘Triple Helix’ Remix and the Role of Open Science 


At this point, it would be tempting to declare that, in spite of the initiatives of nu- 
merous actors, including significant gestures by the government of Quebec, it still 
is “business as usual”. This, however, would be misleading in at least two separate 
ways. First, while it is accurate to say that the Cluster, the Declaration and the 
Observatory all participate in building a certain public perception of everything 
AL, itis not possible to adequate it to an ideology that would hide any sort of naked 
truth.’ In other words, to be critical is to question how the box is made, not to put 
it on fire. Second, the expression ‘business as usual’ undercuts how much the ad- 
vent of deep learning and related AI techniques is changing the power-knowledge 
topography of the province, notably the pivotal role universities are called upon to 
play. A governmentality approach must therefore be attentive to the structuration 
as well as the tensions involved here—which is also to say the historical and geo- 
graphical subtleties that make higher education in Quebec something both North 
American but also profoundly influenced by the French universalistic approach, 


4 Many have indeed noticed how increasingly frequent calls for “ethical Al” from industry figures 
often correlate with ongoing campaigns against “overly coercive” government regulation; see 
both Wagner, 2018 and Greene & al., 2019. A recent variation on this theme seems to be indus- 
try-backed regulations (Simonite, 2019); already, accusations of “regulatory capture” have been 
expressed (Biddle, 2019). 

5 Suchatradition finds an emblematic figure in the early Habermas while he was for instance say- 
ing that “[...] [ideologies] replace traditional legitimations of power by appearing in the mantle of 
modern science and by deriving their justification from the critique of ideology. Ideologies are 
coeval with the critique of ideology” (Habermas 1971, 99). 
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and how, starting in the 60s and 70s, it made a substantial push towards a democ- 
ratization of access.‘ Interestingly, Université de Montréal and McGill University, 
both at the forefront of MILA, are historically considered more ‘elite’ schools while 
still enjoying a great deal of public support. The MILA itself is important not only 
because it attracted most of the government-backed Cluster’s money, but also as 
it comes to embody the displacement and, really, the refinement of what the lit- 
erature calls the triple helix—a schematic model of innovation where corporate 
actors come to mesh with university and government ones (Etzkowitz & Leides- 
dorff 2000)—to now a “quadruple helix” where start-ups, too, are considered key 
strategic partners. Confusing small and large, it is not rare in Montreal to see in- 
ternational corporations such as Microsoft being equated with the local Maluuba, 
Facebook being considered as the emerging FAIR-MTL or Google as an embryonic 
DeepMind—with media celebrating even the smallest of investments.’ All of this 
participates in what we argue is an ecological mentality that blurs the symbolic—a 
hub is positive by its very nature—and the practical, by the virtue of the latest 
trend in what Hoffman and others have called “academic capitalism” (2017; see 
also Slaughter and Rhodes 2010). In turn, the reality corresponds less to the early 
French influence on Quebec’s higher education system than to a mode of “Silicon 
Valley-isation” or “Stanford-isation”, terms borrowed from Salter (2018). 

Common to all AI developments is the fact that they are guided by and insep- 
arable from a specific ethos or model of “open science” (Leonelli 2013; Mirowski 
2018). Researchers see the sharing of information as, prima facie, progress in and 
of itself; discovery and innovation are meant to be picked up by and benefit the 
entire “community” in what is thus an ecological as well as cybernetic mentali- 
ty which, again, has roots in a certain Californian “rebelliousness”.? Today, these 
norms prove to be very efficient, especially with regards to the following three di- 
mensions. First, the obligation to choose to pursue either an academic career or a 
career in private R&D becomes less of an issue when one can publish freely, which 
is now allowed, if not encouraged in most basic research-inclined industrial labs— 
in fact, it is not rare to see papers co-authored by scientist at Facebook or Deep- 
Mind along with university-affiliated researchers. Knowing the shortage of quali- 
fied personnel in AI, such open science practices are thus instrumentally adopted 


6 The ten institutions networked across the territory under the umbrella of Université du Québec 
is emblematic in that regard. 

7 See for instance the summary of investments made to the local ecosystem in 2017 in Mathys 2017. 

8 Here, we want to refer to what Saxenian (1994) and others have described as the characteristical- 
ly innovative way Silicon Valley academic and industry actors had to produce new organizational 
forms at an impressive rate. On many accounts, this distinctive way of establishing collaborative 
ties between actors pertaining to different professional categories but to a common cultural 
background has spearheaded the privileged understanding of how to lead technological innova- 
tion these days—see also Storper & al. 2015. 
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as part of the repertoire necessary to navigate the “war to attract talent” (Hernan- 
dez & King 2016; Metz 2016a; 2017). Second, unrestricted circulation of people and 
ideas should allow for companies to track the best of university research. Intern- 
ships, grants, and philanthropic donations large or small, contribute to secure ac- 
cess to computer science labs and to reach researchers where they are. For a city 
such as Montreal, this has proven very helpful, even if, from this decentralization 
and openness, it is impossible to conclude that its hub is a “plaque-tournante”— 
after all, others like Paris, Singapore, Pittsburg, etc., have benefited too. Thirdly, 
this openness is not only geographical, but temporal, as the adaptation between 
the different helixes, companies and university labs in particular, is intended to 
happen more or less in real-time. The pace of research here is as important as the 
commercial turnover rate that transforms an algorithmic architecture into an 
API, an innovation in a recommendation system, etc. While openness translates 
into windows of opportunity and good timing into fierce competition between 
companies, it is especially important to understand that the logic sustaining the 
entire model really is one of “strategic openness” (Ananny & Crawford 2018). The 
knowledge being produced in universities turns out to be “open for business” in a 
new and understudied sense, especially with regards to its wider implications in 
contexts such as Quebec. 

The fact that deep learning and associated AI technologies signal a substan- 
tial displacement of wealth, prestige and power finds numerous and all the more 
empirical examples to which we will come in a few moments. For now, however, it 
appears that a necessary transition implies to question the broader significance 
of the “exploitable epistemology” (Levy & Johns 2016) set in motion through the 
quadruple-helix and open science nexus. As part of this research, a series of in- 
terviews with individuals involved in AI in Montreal were conducted, with most 
expressing largely consensual views, except for two or three more critical figures. 
The first one came from a computer scientist working in healthcare. Her critique 
points to structural elements in the transformation of research financing in Que- 
bec and Canada: 


It’s a concentration of millions of dollars, it’s as if you're betting on a single number 
at the casino roulette. There’s a variety of different types of research done, not all 

from the deep learning or big data strain [in Al] but that are also innovative—but 
you're not betting on them, you're only betting on deep learning. You're pushing 
everyone in the same direction and you forget that innovation is not necessarily 
of all going in the same direction. You also need to leave some to be sure that re- 
search in its totality is somewhat diversified. That, | see as a threat. It’s going to 

siphon everything in the same direction [..]. In fact, everyone is rushing into it. 
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This comment could serve as a proxy or hint as to how and why even scientific 
institutions partake in the kind of self-fulfilling prophecy that makes deep learn- 
ing a reality. Again, institutions, hype and the pressure towards “Stanfordized” 
research go hand in hand. Another key example would be the attribution of 29 CI- 
FAR Research Chairs late in 2018, some to prominent Facebook associates such as 
Pineault (McGill) or Vincent (Université de Montréal) (CIFAR 2018). For less trendy 
research streams, of course, this draws a path in which difficult access to fund- 
ing would blend with its equivalent in terms of strenuous access to students—at 
least two other computer scientists in public universities from our samples talked 
about how they barely have any grad students nowadays. Looking at the longer 
term, chances are that the situation will become only more cyclical and detrimen- 
tal. 

Another related issue emerging with respect to the meaning of the “exploit- 
able”, even weaponized, epistemology implicated here, concerns the handling of 
databases: who owns them, how are they released, and for what purposes. For 
Big Tech companies as Google, who just open-sourced GPipe, or Microsoft, who 
acquired and now runs Github, gigantic libraries of data are acting in both per- 
formative and legitimating ways. Their flaws and limitations are scarcely if ever 
exposed—the fact for instance that such companies still pursue patents aggres- 
sively (Simonite 2018)—especially when compared to the ecological and cybernet- 
ic benefits attributed to these platforms and widely praised in the media. It is then 
at a more mezzo or local level that things get more challenging. The problem is that 
open data for training is not exactly the same as “real-deal” data or value-added 
data. For instance, our interviewee who works with deep learning applications 
in healthcare insisted that a dangerous dynamic is developing, where start-up 
businesses search for any sizeable bases to access in exchange for deep learning 
services, or at least, make a contract allowing them to share data with a third par- 
ty. In places such as Montreal, to make a profit means finding clients—insurance, 
banks, clinics, biotech, etc.—not yet accustomed to deep learning techniques, ina 
legal environment still unsure about the best way to defend privacy or to regulate 
any potential wrongdoing.’ Yet, it is probably at the micro level of the different 
university labs that the difference between the data “haves” and “have nots” is the 
most striking. Star researchers such as Bengio in Montreal—or, for that matter, 
Hinton in Toronto—attract funding and students because of their close connec- 


9 Animportant parallel should be established with the way failed unicorn Theranos capitalized 
on the biotech industry's important regulatory leeway to position itself as one of the biggest (if 
short-lived) success stories of this emergent field. Its ability to rely on the reputability of ear- 
ly-backers such as Gen. James Mattis, Oracle founder Larry Ellison, media mogul Rupert Mur- 
doch or present-day Secretary of Education Betsy Devos to sustain increasing investment rounds 
should be understood precisely as the result of the field’s relative newness and its lack of proper 
regulations (O’Brien 2018). 
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tion with Google and the like. But what about the other, lesser-known researchers 
in the field? The conundrum is that they almost never gain access to the data ac- 
tually prone to broad commercial applications. To give one final example in this 
section, our team met with another scientist in the summer of 2017 and talked 
about the general sense of community, and what it meant that open science was 
a way for private and public actors in the field to communicate. His answer was 
laconic: “It’s just fake. It’s just fake. They share the algorithm but not the data, you 
can do nothing with this. [As for the meaning of “open” ], it’s just a word because 
I cannot use it”. 


Ill. Deep Learning is Redefining the Private-Public Partnership 


To say that today’s developments in the AI field’s political economy blur the preex- 
isting distinctions between what is deemed private and public—or, for that mat- 
ter, that it amounts to a “Stanfordisation” of higher education in places such as 
Quebec—is not to succumb to any nostalgia for a utopian past. A descriptive and 
agnostic approach is indeed needed to account for, as Hoffman stated, “the com- 
plicated, subtle, and sometimes contradictory ways that commercial logics have 
diffused across academic culture” (2017: 727). The point is that, in Montreal and 
most probably elsewhere, ambiguity is in itself a form of governmentality. Weak- 
ened institutional autonomy is translated into more collaboration; buzzwords in 
the semantic region of “hub”, “clustering”, “ecosystem” and the likes are repeated 
and celebrated in what is then hard to decipher from public relations endeavours 
(see Turkina 2018, for instance). A turning point of this development was the Jan- 
uary 2019 relocation of the MILA to Mile-Ex, a post-industrial inner-city in Mon- 
treal. The relocation of the lab occurred as it got elevated to the status of “Quebec 
Artificial Intelligence Institute” and came to be positioned at the forefront of the 
Mile-Ex’s Cité de PIA, with multiple small and big companies establishing their 
new facilities either in the same building or in its immediate surroundings. O 
Mile-Ex, the converted textile-manufacture the lab moved in, already accommo- 
dated the offices of up-and-coming startup Element AI, Royal Bank of Canada’s 
Al branch Borealis, French military contractor Thales’ AI research division and the 
para-public Institute for Data Valorization (IVADO), with Microsoft’s Maluuba 
also a close neighbour (Bachand 2018; Dubuc 2018). Importantly, the idea to create 
the Cité represents the fourth pillar of the government’s strategy in everything 
Al—along with the Industrial Cluster, the Declaration and the Observatory—yet, 
because of its weight in terms of jobs, investments, square feet of office space and 
the like, it is possible to argue that it is the most important. The people in charge 
there understand rather well the leverage associated with their interstitial posi- 
tion. Indeed, in interviews with media about the relocation, they were keen to ask 
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for additional public funding: “Attracting researchers to Montreal by telling them 
we only have two years of funding left, that won’t work. We need to be part of a 
broader, longer-term vision. We’re in the order of tens and hundreds of millions” 
(Pisano quoted in Rettino-Parazelli 2019). 

In terms of practical, yet non-official public-private blending, there might be 
no equivalent in Quebec to Element AI, the fast-growing company co-founded 
by MILA’s director Bengio. As rightly expressed by one media commentator, “the 
business model is not easy to understand” (quoted in Mercure 2016); not only did 
it attracted historic amounts of venture capital without a proper product on the 
market, but it continually operates under an ethics-oriented discourse of public 
good and social benefits while also positioning itself as an active player in the 
rather traditional and profit-savvy fields of logistics, insurance and banking (The 
Economist 2017). Bengio himself appears willing to play on both levels as he ded- 
icates genuine efforts to promote an ethical and socially-minded development of 
his field while lionizing the commercial success of his company, one apparent- 
ly set to become one of the first Canadian AI Unicorn (George-Cosh 2018; Vara 
2018). In addition, he sometimes confuses his own numerous public and private 
affiliations in talks, Power Points and elsewhere, in what is now emblematic of a 
bigger issue, namely how the value and wealth created in the public domain tends 
to move away from it. The very nature of Element Al—and part of the reason for its 
initial valuation—is to capitalize on its access to star academics to develop ‘busi- 
ness solutions’ for its private-sector clients. As acclaimed in the Journal of Small 
Business & Entrepreneur, the company has “a faculty fellow network composed 
of over 20 world-renowned AI scientists from the top academic labs across Can- 
ada. These professors not only do research-related work for the company but [...] 
provide valuable advice [...]. This unique arrangement gives Element AI access to 
cutting-edge research” (Turkina 2018: 2). Again, what there is in this quotation re- 
lates to everything cybernetic about the new model being implemented discussed 
above. Helixes rotate, openness signals access, pace equals circulation and inno- 
vation, etc., ina movement that is certainly difficult, yet not impossible, to track. 

While Element AI is cybernetic by essence, it is as well ecological in a very 
practical way. Proximity to the MILA shapes the urban space around it, and could 
be measured in meters. Of course, such proximity is not something to be found 
only in the Cité de IA; numerous incubators in North America, Europe and else- 
where use the model with the justification that it contributes to the cross-polli- 
nation of ideas and resources. The problem, however, is slightly different when it 
comes to the blurring of public and private assets. Emblematic in that regard is a 
Facebook post by Element AI saluting the arrival of MILA “to the neighborhood”, 
which showcased a picture of Bengio while emphatically adding, “see you in the 
stairwell” (Element AI 2018). Such metaphor usually refers to a more or less licit 
space; one with more or less fuzzy codes and boundaries. Who goes up, what goes 
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down, when, and how? In the case particularly of students-becoming-interns-be- 
coming-students, the lack of explicit limitations never cease to be problematic 
both on an individual and on a cohort basis. Once aggregated, these public and 
private part-time or twofold affiliations reinforce a model that is poorly checked 
and balanced, especially in the face of its long-lasting socio-political and econom- 
ic impacts and how these could be discussed and amended in the public sphere. 
At current, it is such public-private intermingling that comes to colonize the 
many layers of the AI Montreal hub—despite certain pleas from Bengio against 
a reality he actually contributes to. The recent wave of investments made by 
foreign corporations in the Montreal hub has been going hand-in-hand with the 
increasing adoption by the newly ‘partnered’ scientists of this new organization- 
al arrangement, namely, the dual affiliation model. This university-to-industry 
collaborative form, imported from the fields of law, management and medicine— 
and probably at its strongest in the biotechnology industry; see Mirowski 2012— 
allows scholars to keep their university professorship appointment while adding 
to ita commitment, on at least a part-time basis, to their new corporate employer 
(Serebrin 20172; 2017b). To the list parsed throughout this chapter, we should still 
be adding the many names of MILA-affiliated scientists such as Larochelle at Uni- 
versité de Montréal and Google Brain, Precup at McGill and DeepMind or Pal at 
Element AI and Université de Montréal. Dual affiliation is justified by actors of 
the field as a novel solution where scientists are able to continue teaching and con- 
duct basic research while also participating in industrial R&D, whereas previously, 
such participation would entail a complete retreat from their university teaching 
and basic research activities (Plamondon Emond 2017). The growing dissemina- 
tion of the model thus operates at the junction of two distinct but concomitant 
dynamics. On the one hand, corporate actors are increasingly aware of the neces- 
sity, for their business model, to achieve an all-essential balancing-act between 
the preservation of the “ecosystem sustainability’—i.e., to ensure the continued 
formation of future generations of AI researchers and the further advancement of 
basic research endeavours (LeCun 2018)—and, as described in section II, the con- 
flicting urge of immediate appropriation of specialized human resources (Metz 
2016a; 2017). On the other hand, scientists are responding to constraints which 
are mostly presented as incentives: besides the alluring possibility of alternative 
sources of private funding, researchers also have to deal with the fact that access 
to state-of-the-art corporate computational infrastructures and some of the wid- 
est proprietary databases are indeed technological means increasingly needed for 
the pursuit of cutting-edge deep learning research. In turn, this new public-pri- 


10 See Shead 2018. On his criticizing the increasing concentration by major tech corporations of 
both technological means and specialized human resources while scarcely rejecting opportuni- 
ties to collaborate with them—see Mathys 2017 and Vara 2018. 


135 


Jonathan Roberge/Kevin Morin/Marius Senneville 


vate assemblage and its peculiar way of providing many solutions at once is pre- 
sented, justified and legitimized as a form of necessity, and not as a cascade of 
contingent choices and principles. In that sense, however, it is nonetheless highly 
political and a form of governmentality. 


Conclusion 


This chapter started by acknowledging how the black box is a powerful, yet dis- 
enchanted figure to reflect on technologies in the making such as deep learning, 
and AI more broadly. While there is an urgency that should spur social sciences 
inquiry, it is nonetheless important to do things right, with a certain dose of ag- 
nostic and critical reflexivity. In this light, we attempted to follow Bucher’s triple 
advice to ‘not fear the black box’; ‘not expect the solution to be inside’ and ‘con- 
sider the boxing of the box’. So how did it go? How did the theoretical concepts 
apply to the practical reality and how, in turn, can better understanding of a case 
such as the Montreal hub inform broader and more critical reflection? Parts of the 
answer came in section I, where the argument was made that what is mostly at 
stake is the present and future political economy of AI, i.e. how the automation of 
knowledge production transforms power relations and how the different actors 
involved in deep learning are engaged in what Crandall names a particular form of 
“cooperative struggles” (2010). Substantial resources including money, state sup- 
port, media coverage, etc. are flowing and aggregating, the details of which are 
precisely what must be understood about this dense and tense regime of govern- 
mentality. The new normal brought about by deep learning and Al-related tech- 
nologies will be messy and ambivalent, if this is not already the case. We insisted 
throughout the chapter that power is more than ever a transaction and that what 
“control” means in these circumstances relates to a new sense of cybernetics and 
ecology that shall account for all types of mutualism and parasitism. In section II, 
we described this by digging into the Montreal example, especially how it exhibits 
a peculiar form of rotary motion between the helixes that are the governmental, 
university, established and upcoming corporate actors. Whereas actors repeated- 
ly proclaim there is a “community” and that the Cluster, the Observatory, the Dec- 
laration and the Cité de PIA make for an integrated whole, we propose a somewhat 
less optimistic, more realistic analysis. Open science is a case in point, as “open” 
translates into aggregation and as it signals an important shift in the educational 
model in vogue. There is such thing as a privatisation of higher education in place 
like Quebec, in which deep learning and AI related technologies are instrumental. 
This was the principal conclusion of section III. Whether you call it the “double 
affiliation” or the “see you in the staircase” model, what is clear is that the benefits 
are not equally redistributed at current and have very poor chance—at least the- 
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oretically—of being so in the future. In the end, it might then be such unfolding, 
in its many twists and turns, that constitutes the proper object of another, still-in- 
the-making ecology, one which could be called Critical AI Studies. 
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Three years ago, researchers at the secretive Google X lab in Mountain View, Cali- 
fornia, extracted some 10 million still images from YouTube videos and fed them 
into Google Brain—a network of 1,000 computers programmed to soak up the 
world much as a human toddler does. After three days looking for recurring pat- 
terns, Google Brain decided, all on its own, that there were certain repeating cate- 
gories it could identify: human faces, human bodies and... cats. (Jones 2014: 146) 


1. Deep Learning 


By this point, talk of the omnipotence of algorithms is everywhere. This discourse 
proceeds without interruption and is seemingly impossible to stop—not least be- 
cause algorithms operate quietly and inconspicuously in the background (cf. Bunz 
2012; Seyfert/Roberge 2017). Many of the discussions about their influence con- 
cern the status of their opacity and, by concentrating on the refusal of firms to 
make them transparent, bring arguments into play that seem like relics from an- 
other era. Whereas then the focus of critics rested on the activities of a discredited 
culture industry, today it is the economization of hitherto unimaginable volumes 
of data that is considered a violation. The economic valence of data has become the 
object of a media critique that lost one of its favorite subjects from the previous 
century: the critical and autonomous media user (or that which was once regarded 
as such). The algorithms of large corporations such as Google, Amazon, or Face- 
book rightly seem to have subsumed the latter subject’s potential for action, au- 
tonomy, resistance, and subversion (cf. Sudmann 2017). This process has been so 
successful that it has even led to counter-movements that do not casually lament 
the end of the private sphere as collateral damage of digitalization but have rather 
adopted agendas that enthusiastically promote its undoing (cf. Rieger 2018). For 
the internet exhibitionists of the so-called Post-Privacy Spackeria, data protection 
is nothing more than a historically datable remnant, a vestige from the last mil- 
lennium: “The private sphere is so 1980s.” (Reimann 2019, n. pag.) 

The areas of application for the use of algorithms, which, for their part, have been 
the object of a brief evolution and whose optimization has been oriented not least 
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toward meeting the specifications of nature, are ubiquitous and so varied that they 
cannot be surveyed in full:' whether recognizing faces in everyday life for reasons of 
delayed surveillance or future-oriented forensics, identifying sequences of behavior 
or engaging in biopolitics, clarifying the authorship of images and texts (cf. Rodri- 
guez et al. 2019; Rehman et al. 2019), classifying works of art according to the style 
of a given epoch or comparing signatures supposedly written by the same hand, in- 
tervening in the business of science and confronting apparently non-computable ob- 
jects of knowledge with big data and algorithmization (cf. Rieger 2019), affecting the 
self-perception and self-assessment of certain disciplines over the course of the “com- 
putational turn” and “humanities computing,” associating the latter disciplines with 
different forms of reflection and thereby contributing fundamental changes within 
the humanities itself (cf. Hall 2013), or otherwise intervening in the order of things— 
such activity typically draws upon processes of artificial intelligence, artificial neural 
networks, and deep learning. Their manner of dealing with large volumes of data 
has become a knowledge-promoting game and has even opened up new possibilities 
for Foucauldian discourse analysis, which is seldom applicable to technological de- 
velopments (cf. Engemann/Sudmann 2018). The possibilities of artificial intelligence 
play right into the hands of Foucault’s basic intuition that “empirical knowledge, at a 
given time and ina given culture, did possess a well-defined regularity” and that “the 
history of non-formal knowledge had itself a system.” (Foucault 2002 [1966]: x) Over 
the course of his book The Order of Things, Foucault sought to reveal an epistemologi- 
cally stringent (but, in technical terms, hardly realizable) positive unconscious of knowl- 
edge and thus to give expression to the supposition that there is a “well-defined reg- 
ularity’—a formal code behind non-formal knowledge as well. It would therefore be 
possible to process the science of this knowledge in a different way: it could become 
the object of an algorithmic discourse analysis and remain removed from individu- 
al understanding and comprehension. In the modes of access employed by cultural 
analytics, such a positive unconscious of knowledge is brought up to technical speed 
and made visible in the form of regularities and repetitions. Data mining and text 
mining make patterns and thus forms of knowledge visible that are not necessarily 
exhausted in intentional questions. Here, everything that human intelligence, in its 
scientific narcissism, regards as its genuine field of activity—ordering and classify- 
ing things, identifying similarities, and creating genealogies—is relegated to algo- 
rithms. In this case, the business of science is therefore not at the mercy of chance 
in its efforts to produce knowledge; rather, identities and differences are processed 
automatically—with algorithmic and not anthropogenic support. 

Yet this concerns not only the sciences, with their broad subject areas and the 
claim to complexity associated with them. The activity of algorithms even extends to 


1 The keywords in question would be evolutionary algorithms, evolutionary or genetic program- 
ming. 
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the lower senses, which, for long stretches, received hardly any attention in cultural 

history but have since come into the spotlight thanks to the efforts of various natu- 
ralization movements (cf. Kortum 2008). Like almost everything else, the detection 

of smells can also be delegated to algorithms—with the effect that, where olfactory 
data can be processed automatically in large quantities and at high speeds (in real 

time, to use one of the favorite terms of several protagonists), a familiar danger looms. 
In the case of smells, this danger has been called “odorveillance.” In addition to see- 
ing everything, Jeremy Bentham’s panopticon can now smell everything as well (cf. 
Stark et al. 2018a). The consensus over this seems to be that such a regime of odors 

should be regarded as an outgrowth of other biometric activities and should accord- 
ingly be opposed. Of course, the following is just a rhetorical question: Is this sort of 
odorveillance really what we want? (Stark et al. 2018b: 18) And there also seems to be 

a consensus over the fact that automated activities of this sort should be the object of 
fundamental reflection concerning the nature of “veillance” in all of its varieties (the 

latter now include “sousveillance” and “metaveillance”) (cf. Kammerer/Waitz 2015). 
Indeed, this idea has even been spelled out in a programmatic way—in works with 

titles such as “Declaration of Veillance (Surveillance is a Half-Truth)” (Mann 2015). 


i 
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Fig. 1: Surveillance versus Sousveillance (https://en.wikipedia.org/wiki/ 
File:SurSousVeillanceByStephanieMannAge6.png, accessed June 4, 2019) 
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Protagonists such as Steve Mann (2016) or José van Dijck (2014) should be men- 
tioned here, the former for introducing concepts of veillance beyond surveillance, 
the latter for his concept of datafication, which describes the normalization of 
data politics and its ambit as a new sort of currency. With datafication and its 
basic suspicion concerning the opaque modus operandi of such data processing, 
the media-critical impetus of earlier days seems to have survived and not to have 
capitulated to the demands for a total relinquishment of the private sphere. In his 
book Post-Privacy: Prima leben ohne Privatsphäre, for instance, the internet activist 
Christian Heller comes to appreciate the latter, even though there are arguments 
in favor of its complete abandonment. He cites an example of algorithms being 
able to determine the sexual orientation of individuals from their social behav- 
ior—without any regard, of course, for the safety of the people in question: 

His sexual orientation is private, and so it should remain. However, he creat- 
ed his account without considering the inventors at the Massachusetts Institute 
of Technology (MIT). There a process was developed for identifying, with a high 
probability, the homosexuality of men on the basis of their Facebook profile, even 
if they posted no photos or listed no preferences of any sort. All that is needed is 
to analyze their social environment on Facebook, which is used above all to stay in 
touch with friends, relatives, and acquaintances. Often enough, profiles include a 
list of friends that is visible to anyone in the whole world (it is possible to make this 
information private, but few bother to do so). The researchers at MIT discovered 
that it is possible to make approximate predictions about whether a male student 
is gay on the basis of the portion of men among his Facebook friends who have 
outed themselves as gay on their own profiles (Heller 2011: 12). 


2. Strategies of Participation 


Artificial intelligence is at work everywhere, regardless of whether we know it, 
whether we can know it, or whether we even want to know it. The concerns of sur- 
veillance studies or critical code studies aside, moreover, everyday user behavior 
is often defined by a fundamental and reckless indifference to the activities of 
algorithms and issues of security. As is clear not only from people’s access codes 
and passwords (the use of easily decipherable sequences of numbers, birthdays, 
the nicknames relatives and pets) but also from their willingness to disclose their 
consumer preferences and other habits, this behavior exemplifies flippant and 
careless negligence. Yet there is another aspect that defines what is going on and 
increasingly determines how we engage with artificial intelligence, one that is 
perhaps less visible and at first glance far removed from concrete political action. 
Whereas algorithms are monopolizing autonomy everywhere, whereas they op- 
erate in a self-determined or partly self-determined way, whereas they are exe- 


Reduction and Participation 


cuting the grand scheme of automated knowledge with greater and greater effi- 
ciency and on hardware that is ever increasing in capacity, and whereas—as one 
repeatedly reads—they conduct their business without notice and in the mode 
of operative latency, a peculiar counter-movement is taking place on the level of 
use and participation, social engagement, and the campaign for acceptance. This 
process is peculiar because it seemingly overturns the order of the grand narrative 
that surrounds technology in general and digitalization in particular. The grand 
narrative about the technology around us and the associated politics of the inter- 
net is typically bound to a principle of quantitative growth. This can be narrated 
in the form of large numbers and is written as the history of progress of an utterly 
relentless triumph of increasing complexity. 

It is thus all the more striking to see tendencies in dealing with technical (or 
perhaps it would be better to say socio-technical) infrastructures that move in a 
different direction and are based on the opposite of growth—that is, on what will 
be discussed here under the title “Reduction and Participation.” This interruption 
of the customary success story and the intentional reduction of technically possi- 
ble complexity are noteworthy—and in various ways they revolve around aspects 
of internet politics, democratization, and the question of who should have access 
at all (and in what way). What is especially remarkable is a fundamental expan- 
sion of that which is considered fit for participating on the internet and thus for 
being addressed. Over the course of this expansion, as will be shown, different 
and additional agents have been put in position to participate—agents who are 
situated outside of the dominant concerns of human-computer interaction (HCI) 
and who endorse the argument for reduction or at least provide some indication of 
the gestures associated with it. Those who have somewhat systematically become 
part of the plan include users who, with their specific profiles, veritably embody 
the issue of reduction. These particular users are phenotypically diverse and thus, 
not least, children and people with challenges have attracted increasing attention 
as extreme cases of those with special user profiles: “Alterations of HCI meth- 
ods is common when interaction design is planned for ‘extreme’ human users.” 
(Hirskyj-Douglas et al. 2016, n. pag.) 

Yet this is not just a matter of differentiating human beings according to their 
stages of development (children) or according to their particular challenges (deaf, 
blind, autistic, elderly people; people with cognitive or other challenges). Beyond 
human beings, the aspired reduction of complexity also brings new agents into 
play. Noteworthy in this regard are such things as “animal-computer interaction” 
(ACI). Clara Mancini, one of the leaders of this movement, is quick to point out that 
there is more than just casuistry behind such approaches and that there is more 
to them than mere anecdotes about Skyping dogs and chatting cats (cf. Ritvo/Alli- 
son 2014; Pongracz et al. 2016; Golbeck/Neustaedter 2012). Rather, Mancini’s pro- 
gram stands for a system that is fundamentally related to the field of altered social 
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forms (“interspecies communities”) and is dedicated to promoting “multi-species 
awareness” (cf. Mankoff et al. 2005). She combines her endeavor with the promise 
of an overarching systematic approach and with the self-confidence of a newly 
emerging discipline, as is impressively clear from her manifesto and its positive 
reception (cf. Mancini 2011; Hirskyj-Douglas et al. 2018). 

Not least, this obligation is a matter of social responsibility. As with over- 
stepping the boundaries between species, this is due to more expansive ideas of 
participation (cf. Kelty 2016; Stahl 2014). This attentiveness is accompanied by a 
deeper consideration for the particular features of semiotic systems and by re- 
considerations of one’s own ethical positions (cf. Mancini 2011, 2017). By encour- 
aging the intermingling of species and a political awakening, approaches such 
as ACI are part of a larger intellectual movement known as transhumanism or 
posthumanism. The latter is defined by figures that programmatically renounce 
differentiation. This renunciation is exemplified in Donna Haraway’s book Stay- 
ing with the Trouble: Making Kin in the Chthulucene—especially in her use of the word 
critter, which stands at the center of her thinking. As she notes, this term serves 
as a placeholder for a peculiarly broad range of beings (and machines): “In this 
book, ‘critters’ refers promiscuously to microbes, plants, animals, humans and 
non-humans, and sometimes even to machines.” (Haraway 2016: 169n1) A similar 
argument has been put forth by the philosopher Rosi Braidotti (2013), who con- 
siders all species to be equally vulnerable to the threats of anthropogenic climate 
change and thus urges interspecies collaboration, which, as she vigorously pleads, 
should be part of the political agenda.” It is high time, according to Braidotti, for 
humans to create new social bonds—not only with other species but also with the 
techno-others that we tend to keep at a distance and reduce to their operational 
functionality. Only in such a way does she think it will be possible to ensure our 
common survival as a community facing the same threat.’ What Braidotti pro- 
poses is a fundamental dedifferentiation of the social, which is comparable to the 
dedifferentiation of the ontological in Haraway’s definition (or non-definition) of 
critters. 

Such figures of dedifferentiation, which are central to the theoretical posi- 
tion of post- and transhumanism and thus seek to avoid the habitual accusation of 
anthropocentrism, are necessarily associated with intuitive gestures—a finding 
that unites the numerous movements in favor of openness and expansion against 
the dominance of human-computer interaction. After long phases of political ab- 
stinence, this expansion was joined on the agenda by categories such as respon- 


2 This collaboration should not, moreover, be dictated by a logic of precariousness (cf. Bennke et 
al. 2018). 

3 Such an attitude toward the techno-other is being fostered by a number of anthropophilic ges- 
tures being made on the part of machines (cf. Seaman 2011). 
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sibility, ethics, and participation. The tone of all this is demanding, immodest, 
and programmatic; as Braidotti herself concedes, it is impatient and hardly free 
of pathos. The way in which the concerns of individual participation offensives 
interrelate with those of certain theoretical formulations can be seen, for instance, 
in the work of Fredrik Aspling. The Swedish sociologist is a committed critic of 
anthropocentrism and considers himself a close ally of post- and transhumanism: 


The increased involvement of nonhuman species in interactive contexts supported 
by digital technology, which could be framed as multispecies-computer interac- 
tion, leads to new possibilities and forms of interactions, and consequently, a need 
to reconsider what this is and can be in terms of interaction. (Aspling 2015: 1) 


Multispecies interaction thus becomes the operational basis for a new concept of 
interaction. On this basis, Aspling places a concept of inclusion on the agenda and 
encourages people to consider the particular needs and features of different spe- 
cies: 


The addition of nonhuman species challenges conventional interaction approa- 
ches and theoretical frameworks in HCI. There is a need to think beyond the hu- 
man and confront the challenges associated with the inclusion of other species 
with dissimilar cognitions, experiences, senses, abilities, timescales, wants and 
needs. For further advancement we need appropriate approaches and theoreti- 
cal foundations to better understand the emerging dynamics of these new forms 
of interactions. The attention given to nonhuman species in HCI (e.g., animal as 
legitimate users to design for and with) is in analogy with posthumanism and its 
critique of anthropocentrism. (Ibid; Aspling et al. 2018) 


The issue of going beyond human-computer interaction and integrating new 
agents and processes is part of what is being negotiated by way of concepts such 
as post- and transhumanism and by way of new epochal designations such as 
the Anthropocene or the Chthulucene (cf. Haraway 2016). Alongside gestures of 
ontological opening, which feature prominently in Haraway’s work, there are 
thus also gestures of opening up social interaction. The development of ACI (an- 
imal-computer interaction), PCI (plant-computer interaction), CCI (child-com- 
puter interaction) or RCI (robot-computer interaction) stand for this. The logic of 
subdividing forms of interaction into appropriate departments is just as striking 
as the aspect of promoting all sorts of interspecies collaboration. Interactive re- 
lationships prevail between the knowledge about various individual user groups. 
These relationships make it possible for such groups to learn and profit from one 
another: “The aim is to strengthen connected thinking whilst highlighting the ex- 
changeable connecting methods from both ACI and HCI and their subfields in- 
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cluding Child Computer Interaction (CCI) and Human Robot Interaction (HRI).” 
(Hirskyj-Douglas et al. 2016: n. pag.; cf. Hourcade/Bullock-Rest 2011; Hourcade et 
al. 2018) These interactive relationships and this act of learning from one another 
(“discussing what these fields learn from each other with their similarities and 
differences mapped”) lead to common design criteria. And the latter criteria keep 
the special or extreme user in mind—as children, as people with cognitive or sen- 
sory limitations, as autistic people, and so on (cf. Gennari et al. 2017; Eisapour et 
al. 2018; Lindsay et al. 2012; Satterfield et al. 2016). 

Several of the maxims expressed by proponents “participatory design” are 
syntactical peculiarities. Now it is common to encounter expressions with dual 
prepositions; in order to include special users in advance, for instance, program- 
mers are now encouraged to work for and with them. This double use of preposi- 
tions is important to the movement and therefore often seen. Noteworthy, too, is 
the unusual use of the preposition with. In this context, it is often attached to the 
word becoming, which was one of post-structuralism’s objects of fascination. This 
mode of “becoming-with” (with animals, plants, stones), which concerns both the 
molecular as well as the technical and artificial, is believed to be a key element in 
the struggle for global survival (“the necessity to become-with animals and tech- 
no-objects as a matter of survival” (Davis 2016: 210). 

Cats and children—but also people with challenges, disabilities, or highly 
individual needs—have become the respected target groups of special interfac- 
es made particularly for them (cf. Maaß/Buchmüller 2018; Westerlaken/Gualeni 
2016). Their participation takes place via the reduction of complexity—and this, 
as I have already remarked, in a field that is otherwise defined by gestures of in- 
creasing complexity. Not least, it is defined by a further gesture that involves the 
systematic integration of playfulness; in fact, the impression left is that playful- 
ness is the order of the day and that play itself has a central role in eliminating the 
barriers between species (cf. Nijholt 2015).* Such measures almost make it seem 
as though the professionalization of algorithms is being accompanied by an in- 
fantilization movement—as a sort of counter-movement. This is tied to gestures 
of reduction or can at least be understood under that formula. The programmatic 
nature of the formula owes itself to the discovery that wherever there is talk of 
technology, another narrative is being expressed as well. For such an argument 
in favor of reduction, which is conceived in functional terms and not meant dis- 
respectfully, one should look toward venues that break up and diversify the pri- 
macy of HCI. The recent concentration on children and the efforts—referred to 
by Aspling—to blend child-computer interactions with those of ACI are therefore 
more than mere symptoms: They modulate a praxis of their own. Atypical allianc- 


4 This applies not only to the design of interfaces but also to the design of data and the practices 
associated with it (cf. Anderson et al. 2017). 
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es are now becoming visible and possible, as is evident from the following title of 
an article about ACI: “Of Kittens and Kiddies: Reflections on Participatory Design 
with Small Animals and Small Humans.” (Chisik/Mancini 2017) This organized 
focus on children and cats as representatives of a desired form of intuition ex- 
emplifies some of the concerns of participatory design. The goal is to produce a 
user-friendly interface design that does not have to be laboriously explained but is 
rather intuitive, self-explanatory, and based on tacit knowledge. Participatory de- 
sign is negotiated both with as well as between humans, animals, and machines. 
With its focus on small animals and people, it makes reduction tangible. What is 
more, it makes reduction the keystone of participation. 


3. Asymmetries 


The naturalization of designs meant for interaction, collaboration, or commu- 
nication requires the use of surfaces and has operative dimensions (cf. Norman 
2010). Thus it is not exhausted by gestures of dedifferentiation but rather goes 
hand in hand with strategic considerations. One of these is the discovery of the 
multisensory—or, as Caon et al. (2018) have called it, “multisensory storming.” 
Storming the senses has been able to take place, first of all, through the increasing 
discovery of the tactile and the haptic—a discovery over whose course the man- 
ners of speaking about computers and algorithms have themselves been changing. 
Gestures of naturalization, which have been described as well as criticized within 
the discussion about interfaces, concern not only the problems of dealing with 
hardware but also manners of programming (cf. Bruns 1993; Hornecker 2008). Not 
only does the computer require massive strategies for accommodating the sens- 
es; the activity of programming is also under pressure to recreate itself in a new 
image. It has to abandon its cognitive solipsism and, beyond merely working with 
symbols, become a tactile undertaking. Thus, yet again, the body will become the 
natural guarantee of a form of comprehensive participation that can or should 
be able to take place without effort, intuitively, and in the transparent mode of 
self-evidence. 

The issue is not only computer use and literacy but also a life world that al- 
lows technology to exist in any given ambient form. By now there are abundant 
examples of this and, on a systematic level, they tend to have certain features in 
common, most notably the development of new channels, the integration of dif- 
ferent senses, and the emergence of new forms of communication. The latter free 
up scenes of asymmetrical communication—scenes that invalidate the common 
conceptions of communication theory. The abundance of examples extends from 
applications for remotely caring for pets to interacting with plants, which are of- 
ten grown in artificial environments (cf. Lee et al. 2006; Kuribayashi et al. 2007). 
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They alter forms of sociality. One of the most theoretically ambitious protago- 
nists in this field is the Japanese researcher Hill Hiroki Kobayashi.” His goal is to 
transcend a paradigm of communication and interaction that is measured solely 
on the basis of human beings (in full possession of their mental faculties) and a 
particular form of linguistic communication. Kobayashi’s notion of “human-com- 
puter-biosphere interaction” (HCBI) has a virtually unlimited field of operation. It 
not only changes the sphere of actors but also, and necessarily, the ways in which 
communication takes place: “HCBI extends the subject of Human Computer In- 
teraction (HCI) from countable people, objects, pets, and plants into an audito- 
ry biosphere that is uncountable, complex, and non-linguistic.” (Kobayashi 2010: 
n. pag.) This abandonment of the anthropocentric standpoint is as much a pro- 
gram as it is a collaboration with agents that elude the principle of countability 
(cf. Kobayashi 2014). In this way, possible forms of expression beyond articulated 
speech are assigned a central role. Regarding the use of wearables that are meant 
to bring people closer to nature (“Wearable Forest-Feeling of Belonging to Nature” 
is the title of his article), Kobayashi writes: “Thus, wearable computer systems 
have become an inter-medium to express the telepresence of various species in 
the biosphere in such a way that their non-linguistic expression is perceived and 
understood by each participant, which violates all the rules of linguistic science.” 
(Kobayashi 2008: 1133) 

The locus for such applications is thus close to life and by no means limited to 
art installations. An indication of how lifelike they can be is provided by a device 
called LumiTouch. At first glance, LumiTouch looks like a regular pair of picture 
frames. One inconspicuous frame is connected to an equivalent through the in- 
ternet, and it is able to trigger signals that correspond to someone’s mere touch. 
Depending on the type of touch (its intensity, frequency, duration), various light 
patterns and color constellations are released that can be associated with an indi- 
vidualized code. According to its designers, the latter is suitable for implementing 
a special form of expression and thus encourages the development of a private 
emotional language (cf. Kaye/Goulding 2004). LumiTouch changes the simplified 
(because idealized) models of communication theory, and the act of touching the 
picture frame has useful advantages for people with impairments. What its de- 
signers envision are forms of asymmetrical exchange for which one of the com- 
munication partners does not need to be in full command of his or her cognitive 
or physical abilities: “People who are unable to actively communicate for long pe- 
riods of time (e.g. sick or elderly) might be able to use the passive transmission of 
LumiTouch.” (Chang et al. 2001: 314) The potential of overtaxing motor skills or 
cognitive faculties in certain situations, such as when someone is bed-ridden, can 
be counteracted with communicative systems that are less demanding: “Similar- 


5 See his homepage at http://hhkobayashi.com (accessed June 2, 2019) and Nijholt 2015. 
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ly users who lack the required dexterity or concentration for pushing numerous 
buttons might appreciate this system due to its small number of simple grasping 
inputs.” (Ibid.) 


Fig. 2: LumiTouch (Chang et al. 2001: 314) 


Another system that is based on reduction is a product called Tsunagari-kan’ Com- 
munication, which is devoted to the goal of ensuring communication between 
distant family members (cf. Miyajima et al. 2005). Here, too, what is favored is 
a non-linguistic form of intimate communication (“Tsunagari’ communication 
aims to foster a feeling of connection between people living apart by exchang- 
ing and sharing the cue information via network everyday.” (Itoh et al. 2002: 810) 
Expanding upon LumiTouch’s model, it also allows communication to take place 
in the mode of the unconscious and passive. Using a so called “Family Planter” as 
a communicative tool, it is meant to enable firm social bonds to form through 
exchanges of “cue information” (ibid.: 811). By means of infrared and ultrasound 
sensors, Tsunagari’s interconnected terminals react to a person’s presence and 
movement. This information is transmitted and converted on the receiving end 
into a non-linguistic signal: 
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Optical fibers at the top of the terminal will gleam to indicate the remote human 
presence and will rotate to indicate the remote human motion. This is intended to 
exchange presence and movement information implicitly (without explicit inter- 
vention from users) and constantly. (Ibid.) 


Optical Fibers 
(bright and rotating ) 


Terminal 


Touch 
Sensor 


Infrared Sensor 
Ultrasonic Sensor 


(motion detection) Terminal 


Fig. 3: Family Planter (Itoh et al. 2002: 810) 


The design of this planter-based sensory device contains several important as- 
pects that also happen to be central to interspecies communication and interac- 
tion. In the mode of implicit and thus unconscious participation, the system al- 
lows people to partake in the everyday lives of remote family members seamlessly 
and in a way that is not felt as an imposition or disruption: “These exchanges are 
designed to blend into the everyday life of a user.” (Ibid.) By means of three sen- 
sors, it can transmit various audio signals, and thus the system can also be used to 
convey explicit messages. More important than this explicit mode, however, is the 
implicit nature of its use, which, with its unobtrusive participation, submits to the 
logic of media and the way in which they increasingly blend inconspicuously into 
our environments. They now do so seamlessly, unobtrusively, quietly, smoothly, 
and ubiquitously, and these qualities are redefining the ways that theorists should 
think about media in general. This would be a media theory that, freed from the 
paradigmatic idea that media are extensions or organic projections of humans, 
could instead be described with adjectives such as ubiquitous, seamless, and calm 
(cf. Weiser/Brown 1996). It would be a media theory that directs its focus toward 
the issues involved with making communication more intimate and embraces its 
own intimately charged objects. This trend toward developing things that can be 
laden with affect is only growing. It is driven by an identifiable agenda and not 
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by casuistry (cf. Choi et al. 2014; Kaye/Goulding 2004). Its basis—reduction—has 
become a program whose results will become a part of everyday life. 


br 


Fig. 4: Lamp (Angelini/Caon. 2015: n.p.) 


4. Finis (hominis) 


Children and cats aside, what all of this brings to light are the needs and venues 
of a sort of communication and collaboration that is designed to be asymmetri- 
cal and yet non-discriminatory. The applications presented above do not aim to 
optimize ways of dealing with technical environments but rather hope to provide 
alternative and less complex ways of using them (cf. Rieger 2019). Thus the view 
has also shifted away from the previous stubborn orientation toward a particular 
type of user (cf. Satchell/Dourish 2009). Two things remain to be said in closing: 
First, the children and cats, which I have introduced here as representatives of a 
broader phenomenon, are being put to functional use. What this comes down to is 
not an offer of minimization, such as that which defines rampant cat content, but 
rather the functional equivalent of a strategically pursued reduction of complexi- 
ty. Among these pursuits are campaigns for acceptance that include special users 
and shift the focus of designs toward all possible forms of participation. One of 
the latter is the gesture of naturalization (cf. Andreas et al. 2018). 
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The second point concerns the question of who rules the network. To this ques- 
tion there is, at first glance, a simple answer, and it has nothing to do with the 
power of inconspicuous algorithms but rather with online content. It was none 
other than the deep-learning processes of Google Brain that brought to light the 
fact that it is cats that have, in quantitative terms, been dominating what is going 
on there (cf. Guerin/Vasconcelos 2008). Much to the amusement of those work- 
ing on the project, their algorithms revealed that, indeed, the cat is the lord of 
the internet—a supposition that Alexander Pschera (2016) also plays with, though 
somewhat less jokingly, in an article devoted to the “internet of animals.” For some 
time now, the internet has not belonged to people alone. This situation is now even 
reflected in puns that, as silly as they may be, nevertheless support the ethical 
arguments of participatory design: “Our work focuses on canine companions, and 
includes, pawticipatory design, labradory tests, and canid camera monitoring.” 
(Mankoff et al. 2005: 253; cf. Trindade et al. 2015) Or, regarding cats in particular: 
“In the modern era of digital media, it is hard to deny that cats have clawed their 
way into the zeitgeist of the Internet.” (Myrick 2015: 175) 

The title that I have chosen for this essay— “Reduction and Participation’”— 
takes the demands for including other species and forms of existence at their 
word. The aim of such demands is to expand the circle of those with agency and 
epistemic relevance. Multispecies communities will be home to new actors, new 
forms of communication and collaboration, new types of design and participation, 
new responsibilities and social forms: between humans and animals, plants and 
stones, artefacts and biofacts, machines and media, the living and the non-living, 
the real and the virtual, the augmented and un-augmented, the simulated and 
the modelled, the increased and the reduced (cf. Leistert 2017). It is therefore only 
consistent that, in this sphere of actors, algorithms might not find their peace but 
will certainly find their place. 

Translated by Valentine A. Pakis 
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The Political Affinities of Al 


Dan McQuillan 


Introduction 


We need a radical politics of AI, that is, a politics of artificial neural networks. AI 
acts as political technology, but current efforts to characterise it take the form of 
liberal statements about ethics. Issues of bias in AI are treated as questions of fair- 
ness, as if society is already a level playing field that just needs to be maintained. 
Transparency and accountability are seen as sufficient to correct AI problematics 
(ACM FAT 2019), as if it is being introduced into well-functioning and genuinely 
democratic polities. The apparent refusal to see Al as political flies in the face of its 
promotion as a solution to austerity. In the UK, for example, discussions about the 
under-funded public healthcare system are peppered by senior level statements 
that “AI may be the thing that saves the NHS” (Ghosh 2018). Austerity is not a natu- 
ral disaster but a political decision to prop up financial institutions at the expense 
of public spending. The hope of those decision makers is that machinic reasoning 
can solve the riddle of dealing with rising needs using sharply reduced resources. 
Meanwhile the operating characteristics of actual AI have other political impacts, 
such as the deracination of due process. The vast parallel iterations carried out by 
backpropagation cast an opacity over AI by making its optimisations very hard 
to reverse to human reasoning (Lipton 2016). Algorithmic judgements that affect 
important social and political decisions are thus removed from discourse. 

The political dimensions of artificial intelligence cannot be divined in the ab- 
stract nor solved by philosophical ethics. They result from concrete technical op- 
erations, such as sums over vectors, in the context of specific social conditions. 
The idea of ethical AI is an information operation designed to calm public fears 
about algorithmic impacts, and to position it for market advantage (Hern 2018). 
The real hazards of AI emerge as it intermingles with the political currents of our 
time. A figure for the political entanglement of AI is a photograph taken at the 
recent World Economic Forum showing the populist and extreme right Brazilian 
politician Jair Bolsonaro seated at lunch between Apple CEO Tim Cook and Micro- 
soft CEO Satya Nadella (Slobodian 2019). Artificial neural networks are in demand 
because the confluence of big data and processing power, in the form of GPUs, has 
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enabled them to produce uncanny results in fields like image recognition. How- 
ever, the years in which AI is coming of age are also the years of neoliberal crisis 
and a global rise in far right politics. The urgent question is, how do the concrete 
technical operations of neural networks reinforce, enforce or extend these politi- 
cal currents, and how (if at all) they might instead serve the goals of social justice. 


Boundaries 


The attraction of deep learning is its ability to produce predictions from overflows 
of data. The weights in the layers are optimised by iterations that drive the loss 
function into a minima, while activation functions like ReLU act ruthlessly at 
each neuron to remove weaker signals (3Blue1Brown. n.d). The overall effect is 
the production of statistical certainty; a net of weights that will transform messy 
input data into unambiguous classifications. This is done by substituting correla- 
tions for any attempt to establish causal mechanism, and is not constrained by 
any wider framework of consistency. There is no element of ‘common sense’ in the 
mechanism that differentiates between guesses based on embodied experience 
of the world. Neural networks are neoplatonic; they claim a hidden mathemati- 
cal order in the world that is superior to direct experience (McQuillan 2017). The 
politics enters in the way these orderings are entrained in wider mechanisms. In- 
stead of constraining statistical authority based on a broader care for the human 
consequences, the current race to adopt Al is driven by the way its single-minded 
optimisation resonates with institutional goals of maximising efficiency or share- 
holder value. The operations of AI act in harmony with a neoliberalism that per- 
ceives the world as an atomised set of inputs into a market mechanism that will 
necessarily produce the optimum result. 

The purpose of Al’s mathematical regressions is to draw decision boundaries, 
such that an input is cleanly categorised as on one side or the other. Connecting 
this to matters of risk in the external world, even where the ends are supposedly 
benign, propels AI into being a system of control. Its calculative categorisations 
trigger chains of machine and human decisions with real consequences, involving 
the allocation or removal of resources or opportunities. Embedded in deep learn- 
ing, obfuscated from due process or discourse, these numerical judgements have 
a law-like force without being of the law. Thus, the predictive boundaries of Al 
map outwards as continuous partial states of exception (Agamben 2005). The ex- 
pertise to contest the calculations of machinic reason in their own terms is highly 
centralised in a few corporations and universities. For the rest of us, the calcula- 
tive authority of machine learning leads to situations where personal testimony is 
devalued with respect to computational insights. AI becomes an engine for epis- 
temic injustice, claiming insights that override lived experience (Fricker 2007). 
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Bureaucracy 


Like bureaucracy in the twentieth century, AI is poised to become the unifying 
logic of legitimation across corporations and government. At the current time, the 
performance of deep learning is proportional to the amount of computing pow- 
er used: between the AlexNet image recognition breakthrough of 2012 and the 
Google DeepMind system that beat the Go grandmaster, the required processing 
power grew by a factor of 300,000 to around 2000 Petaflops/s-day (Amodei/Her- 
nandez 2018). The hardware and software pipelines of deep learning are becom- 
ing strategically important, and existing instances like Amazon Web Services are 
increasingly indistinguishable from critical national infrastructure (Konkel 2016). 
But although AI is materialised in the fenced-off anonymity of server farms, its 
leverage lies between thought and action. Deep learning applied to social decisions 
becomes the concrete manifestation of Bourdieu’s habitus; structured structures 
predisposed to function as structuring structures (Bourdieu 1990). It is not that 
key decisions are delegated to machines with no human in the loop; rather, that 
people making pressured decisions are presented with empirical rankings of risk, 
who’s derivation they have no way of questioning. AI encourages thoughtlessness 
in the sense described by Hannah Arendt; the inability to critique instructions, 
the lack of reflection on consequences, a commitment to the belief that a correct 
ordering is being carried out (Arendt 2006). 

Through prediction, this ordering extends bureaucratic governmentality to 
the domain of intent or tendency, which it strives to preempt as a service or as an 
intervention. As well as classifications of pre-crime and the proliferation of forms 
of ’pre-extremism’, as prototyped by the UK’s Prevent Strategy (Sian 2017), there 
will also be classifications that claim benevolence and efficient resource allocation, 
such as pre-diabetes or pre-dementia (LaMattina 2016). The drive for preemption 
enfolds social fears and market interests with the aim of eliminating that which 
is undesirable. The problem is that AI is reductionist. It can only learn from those 
aspects of the context that can be mathematised, and it is given a singular goal 
to optimise on. Therefore attempt by Alto explain what is going on reduces the 
entire system to certain constituent elements and their interactions. Moreover, 
predictive deep learning applied to social questions implies that attributes are 
individualised and innate, while obfuscating the background of common social 
causes. It will extend an apartheid bureaucracy to any aspect of life touched by 
data. 
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Instability 


AI should not be applied to any part of complex social and cultural problems, out- 
side of extremely narrow and restricted aspects. This is not only because its mode 
of operation encourages thoughtlessness and reductionism, but because deep 
learning is literally out of its depth when it comes to social and political complex- 
ity. It is, in essence, simply a pattern finding technique which works surprising- 
ly well at perceptual classification and in some other well-bounded applications 
such as game playing. However, even in these heartlands of AI there are signs 
of systematic problems. There are many adversarial examples where the addition 
of carefully chosen noise to an image, which appears to human perception as no 
more than a scattering of insignificant white dots, can force a neural network to 
wrongly classify an obvious image (Goswami et al. 2018). Perhaps more signifi- 
cantly, a recent paper shows that deep learning’s image recognition often falls 
apart when confronted with common stimuli rotated in three dimensional space 
into unusual positions. In one of the examples, the network correctly recognises a 
school truck, but when it sees a real picture of one on it’s side it mis-classifies it as 
a snow plough. The authors conclude that, while deep neural networks work well 
at image classification, they are still far from true object recognition, and their 
understanding of objects is quite naive. Their conclusion is that “deep neural net- 
works (DNNs) can fail to generalize to out-of-distribution (OoD) inputs, including 
natural, non-adversarial ones, which are common in real-world settings” (Alcorn 
et al. 2018). This is an important but hardly surprising observation. No neural 
network has any understanding of anything, in the form of an abstract model or 
ontology that can be freely applied to novel situations. That is, neural networks 
are incapable of exactly the kind of adaptive and analogical thinking that char- 
acterises even young children. Statements from leading AI engineers that neural 
networks would either “now or in the near future” be able to do “any mental task” a 
person could do “with less than one second of thought” (Ng 2016) is not only laugh- 
able but actually dangerous. If deep learning can’t recognize objects in non-ca- 
nonical poses, we should not expect it to do everyday, common sense reasoning, 
a task for which it has never shown any facility whatsoever. Still less should we 
apply it in messy socio-political contexts and expect it to draw out insights that 
have previously been delegated to discourse. 

However, being out of its depth is not the only reason we should keep deep 
learning clear of socially sensitive situations. The single-minded optimisation 
that makes AI resonate so well with a neoliberal perspective brings with it a fatal 
ethical payload. Utility functions, like deep learning’s backpropagation, get in to 
ethical deep water when there are independent, irreducible objectives that need to 
be pursued at the same time. Ethicists have theorems that suggest it’s impossible 
for an optimisation to produce a good outcome for a population without violating 
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our ethical intentions. For example, the mere addition paradox shows that, if opti- 
mising on a social welfare function over any population of happy people, there ex- 
ists a much larger population with miserable lives that is ‘better’ (more optimised 
for total wellbeing) than the happy population (Eckersley 2018). Not surprisingly 
this paradox is also known as the repugnant conclusion. While this may seem to 
derive from an abstract, analytical logic of moral philosophy, let us remember that 
that is the point: through institutionalised neural networks we are applying an 
abstract and calculative logic to the social world. Similar ethical reasoning has 
produced a whole set of unappealing paradoxes such as the ‘sadistic conclusion’ 
and the ‘very anti-egalitarian conclusion’. These suggest a basic incompatibility 
between different utilitarian objectives such as maximizing total wellbeing, max- 
imizing average wellbeing, and avoiding suffering. Thoughtless pursuit of an ob- 
jective function, as instrumentalised in AI, leads to ethically toxic consequences 
even when the initial function is apparently benign, let alone when it serves the 
capitalist goal of profit. 


The possible 


Thoughtlessness also enters at the start of the road to an AI solution. Al is always 
in the service of solving what Bergson called ‘ready-made problems’. That is, ma- 
chine learning is applied to problems which are based on unexamined assump- 
tions, such as cultural biases and institutional goals, and those deeper prejudices 
which are embedded in language itself. The problem with a ready-made problem 
is that it presupposes a range of possible solutions which are coterminous with 
that particular expression of the problem. Bergson argued that if one accepts a 
ready-made problem “one might just as well say that all truth is already virtually 
known, that its model is patented in the administrative offices of the state, and 
that philosophy is a jig-saw puzzle where the problem is to construct with the 
pieces society gives us the design it is unwilling to show us.” (Henri Bergson, La 
pensée et le mouvant, cited in Solhjdu 2015). To have agency, to be able to change 
a given reality, is instead a question of finding the problem and of positing it. This 
is different because “stating the problem is not simply uncovering, it is inventing. 
According to Solhdju, Isabelle Stengers expresses this as “the difference between 
the possible and the probable” where the probable is “that which with respect to 
the real only lacks one single thing, existence” (Solhdju 2015). It can already be 
constructed using the same conceptual scaffolding that was used to build the 
problem, and figuring it out is simply a matter of probabilistic deduction. The 
possible, on the other hand, is of something unpredictable and non-calculable; a 
creative act that is not merely rearrangement of existing truths. What’s at stake is 
not the probable of current AI but the possible of political thought and action. We 
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need to approach AI in a way that enables us to take sides with the possible against 
probabilities. 

Recreating the possibilities of machine learning means working with pro- 
gramming and politics as non-divisibles, solving engineering problems while 
sustaining a focus on social impacts. It requires both precision at a mathematical 
level and and openness towards the different possible realities that might be ar- 
ticulated. One approach to such a discipline might be offered by a feminist model 
of science, such as that described by Roy, Harding and Spanier (Roy 2004). That is, 
an expanded form of scientific methodology that includes the origination of the 
problem and the purpose of the inquiry. Those wishing to develop non-oppressive 
machine learning should not accept a problem as given, but should start by locat- 
ing its origins, in other words the structural forces which have posited it and pri- 
oritised it. Uncovering the purposes of an inquiry with deep learning means going 
beyond accurately predicting the validation data by optimising hyperparameters. 
It means understanding this narrow technical purpose as part of a broader set of 
impacts, asking who’s ends it will serve, who it might exclude, and how it would 
effect the wider wellbeing of society. Perhaps most radically for AI, a feminist ap- 
proach establishes a relationship between the inquirer and their subject of inquiry, 
requiring us to purposefully put aside the onlooker consciousness that fuels Al’s 
hubris. The most direct way to put this feminist method into practice with ma- 
chine learning is through collective structures of research that include the ‘target 
group’ in the process of inquiry, through structures such as people’s councils (Mc- 
Quillan 2018). Such situated collectives of inquiry are well placed to re-invent the 
problem as lines of flight from the tyranny of the probable. 


Political action 


We must establish this alternative against the political currents that resonate with 
thoughtless AI. This will not be an easy task. AI being implemented as ‘AI under 
austerity’, that is, as neoliberalism’s response to its own crisis. Everyday cruelties 
such as welfare cuts to the disabled are being increasingly obfuscated by machinic 
classifications (Alston n.d.). Neural networks could become engines of epistemic 
injustice and partial states of exception. Even more dangerously, the simplifica- 
tion of social problems to optimisation based on reductionist reasoning and in- 
nate characteristics echoes exactly the politics of the populist far right. Pointing 
out the inconsistencies in the claims of AI has no traction with this political ten- 
dency. Stupidity and hate don’t require philosophical consistency, only an opera- 
tional effectiveness that performs their ideological theatre of cruelty. The practice 
of AI must develop a politics that resists authoritarianism and asserts a care for 
our common humanity. 
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Thus the necessity of collective practices of AI is not only an epistemological 
necessity but a political one. The political forms of the people’s council and the 
general assembly can return the questions of due process and justice to their 
proper place in discourse. Where algorithmic authority comes from privileging 
generalised abstractions, direct democracy can be reasserted by the mobilisation 
of situated knowledges. These need to be channelled into forms applicable to com- 
putational technologies. Ivan Illich, in his call for convivial technology, proposed 
‘counterfoil research’ whose goal is to detect “the incipient stages of murderous 
logic in a tool” (Illich 1975) where a tool, for Illich, means a specific combination 
of technologies and institutions. Counterfoil research lays out a plausible pro- 
gramme for AI people’s councils; that they should “clarify and dramatize the rela- 
tionship of people to their tools”, “hold constantly before the public the resources 
that are available and the consequences of their use in various ways” and identify 
“those classes of people most immediately hurt by such trends”. This is not a neg- 
ative programme but a positive one, to create conditions where people have the 
capacity for autonomous action by means of tools least controlled by others. The 
goals is to find appropriate limits for our tools. Limiting tools through the mech- 
anism of assemblies also creates what Hannah Arendt identified as spaces for ac- 
tion, which only arises from face-to-face encounters and is that which happens 
“against the overwhelming odds of statistical laws and their probability” (Arendt 
1998). 

However, such spaces will not be freely given. Forms of resistance will be 
necessary to create them. One potential form of resistance is in worker self-or- 
ganisation, both in the heart of AI engineering and in the other places of work 
which will be affected by it. There are small signs of the former in way employees 
of corporations like Google, Microsoft and Amazon have expressed dissent at the 
adoption of their technical creations by the military and security apparatus (Alba 
2018, Conger 2018, Lee 2019) whereas the latter has so far been limited to those 
precarious workers, like Uber and Deliveroo drivers, who are already “below the 
algorithm” (Möhlmann/Henfridsson 2017). In workers self-organisation, too, the 
collective forms of the assembly and council have a key role to play, especially in 
advancing the ambition of workers to know that “by organizing industrially we 
are forming the structure of the new society within the shell of the old.” In the 
mid-1970s workers in a major arms company used grassroots assemblies to gen- 
erate a plan for restructuring their factories. Their programme, the ‘Lucas Plan’, 
would have not only converted the activity of the machinery away from arms 
production but included newly invented possibilities for products which, in ret- 
rospect, seem ahead of their time in terms of environmental impact (Open Uni- 
versity 1978). Another potential form of resistance that may emerge by necessity is 
Luddism, where people oppose the predations of hegemonic technology through 
direct action. The historical Luddites’ opposition to steam-powered machines of 
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production was based on the new social relations of subjection that they produced. 
Rather than some atavistic dislike of technology, the resistance of the Luddites 
was motivated by their alternative social vision (Binfield 2004). Their call was to 
‘put down all Machinery hurtful to Commonality’. A new Luddism is one way to 
characterise attacks on self-driving vans by residents in Arizona, fed up of the 
way Waymo is testing its autonomous Al in their communities and on the streets 
where their children are playing. Deep learning has proved again what the radi- 
cals of the 1970s claimed, that the domain of production has extended to everyday 
life, and that we live in the ‘social factory’ (Cuninghame 2015). 

Historical Luddism was part of a wider uprising of workers and communities 
that deeply rattled the emerging industrial elites of its time. AI as it stands is the 
tool of a new technocratic elite. Whatever the strategies for restructuring AI, they 
clearly won’t come about without engagement with the wider field of progressive 
politics. AI, in the form of neural networks, is an inherently political technology 
which must be acknowledged as such. Adopted without constraints it will tend to 
amplify the injustices of the status quo, or even become part of a shift to a darker 
normativity under the hostile environment of the far right. There is, however, the 
possibility of an AI that consciously aligns itself with ideals of social justice and 
egalitarianism. Not as autonomous decision making, but as part of a movement 
for social autonomy. This is Al as part of a wider structural renewal, supporting 
the withdrawal of power from hegemonic institutions and the creation of alter- 
native structures of social organisation based on mutual aid (Landauer/Richard 
2010). Reclaiming our own agency is not to attack AI as such but to challenge the 
system that produces Al in its own image. This is what it means to take sides with 
the possible against the probable. Retrieving our capacity to think collectively, 
learning from and with each other rather than relying on machine learning, we 
can counter thoughtlessness with practices of solidarity and collective care. 
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Artificial Intelligence 
Invisible Agencies in the Folds of Technological Cultures 


Yvonne Forster 


1. Introduction 


Democracy is all about transparency, visibility, and public engagement. In the 
Greek polis, political decisions were discussed in the agora, a public place where 
all citizens (in that case only free men older than 30) could listen and engage. Rep- 
resentational democracy today is less public, but transparency of decision pro- 
cesses is of the utmost importance. If a government cannot make its decisions 
transparent enough, it runs the risk of losing the people’s trust. Transparency in 
a political sense implies rules, visibility, and the readiness to argue and give rea- 
sons. With the emergence of AI applications not only in the political sphere but 
in basically every aspect of social and private life, we are faced with new forms 
of opacity and nonconscious cognition, which strongly impact human decision 
making, behavior, movement, and communication. The central problem is that AI 
applications act without being able to give an account of the underlying reasons 
and even the underlying causal processes remain opaque (black box). Ifan Al used 
for analyzing credit rating denies credit, this decision can ruin a private life. If 
then reasons are not given or possibilities explained, this alone might shake peo- 
ple’s trust in civil society. Agency based in nonconscious cognition is becoming a 
ubiquitous phenomenon and thus calls for ethical and phenomenological reflec- 
tion. In this essay, I aim at understanding the way in which Al is experienced in 
terms of visibility and transparency. Toward this end I will combine phenomeno- 
logical considerations with Martin Heidegger’s reflections on the nature of tech- 
nology. 

One of the features that elicits speculation about artificial intelligence at stake 
here is the fact that at least for the user it is nearly impossible to understand how 
Al arrives at its outputs. AI applications are often characterized as black boxes 
(cf. Sudmann 2018a). Even if the math behind self-learning algorithms is quite 
straightforward, the causal processes leading from input to output are not really 
transparent (cf. Sudmann 2018b: 63). Obscurity is usually conceived of as a threat 
and potential danger. This leads to the central question of this article: Should Al 
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be regarded as a threat to democracy because of its invisibility? As I will argue, 
this is true at the surface, but I will also show that technology always comes with 
a certain form of invisibility. The question is whether this reaches a new level with 
AI. Ina first step, I will define what I mean by visibility/invisibility from a phe- 
nomenological perspective. I introduce this view because it relates perception, ex- 
perience to technology. Then I will clarify how this applies to the relation of human 
and artificial intelligence. The last part of the paper discusses the issues of the 
disappearance of technology and the complex relation of transparency and opac- 
ity with regard to technology. My aim is to show how AI systems introduce a new 
kind of invisibility or opacity to the ecological structures of the life-world. 

There are at least three different layers in the interplay of visibility and invisi- 
bility involved: One goes for every object of perception: Perception is perspectival 
and thus invisibility is a necessary part of it. Invisibility therefore is a constitu- 
tive part of every form of perception and cognition. In the case of technology, I 
follow Heidegger in the diagnosis that there is a higher order form of invisibility. 
This is the essence of technology, which is itself not technological, but a funda- 
mental style of thinking or revealing. This analysis of technology has a parallel in 
the analysis of consciousness, which is in its constitution also opaque to the con- 
scious subject. To this extent there is nothing groundbreaking or new in terms of 
technology. With AI a third layer of opacity enters the stage: This is nonconscious 
agency—an agency that cannot give reasons but shapes lives in a very profound 
way. Although nonconscious agency is present also in humans and animals, tech- 
nological nonconscious agency is new because it essentially shapes social and po- 
litical life now and in the future. The combination of these aspects of invisibility 
and opacity makes up for the widespread uneasiness with AI. My aim is to give an 
idea how the different forms of visibility, transparency, and opacity influence the 
potential of AI to endanger or enable democracy. 


2. Conditions of Appearance: Visibility and Invisibility 


In his essay, The Question Concerning Technology, Martin Heidegger describes tech- 
nology as a way of revealing, of bringing the concealed into unconcealment (cf. 
11f.). This view is more profound than the usual instrumental view of technology 
as a means to an end. The character of technological artifacts is not understood 
adequately according to Heidegger, if this is conceived of as a tool that simply 
helps humans achieve particular ends. Furthermore, Heidegger also claims that 
seeing technology as a human doing does not capture it fully. Both notions of 
technology as instrumental or anthropological are not wrong. They capture tech- 
nology in terms of how it is usually experienced and used. Nevertheless, they do 
not get to the essence of what technology is. But what is the essence of something? 
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Is it the thingness of a thing, that through which a thing is a thing? Is it something 
that does not change, while other parts or aspects may do so? In fact, it is hard to 
specify conceptually what the essence of something actually means. 

In Heidegger’s writings, at least two notions of the concept are at stake: First, 
the ancient Greek notion what something is (Heidegger 1977: 4); and, second, that 
of “enduring as presence” (Heidegger 1961: 59). Both aspects are relevant in his 
essay. The quest to understand what technology is determines the whole text. 
Heidegger is convinced that the answer to this question will not point towards an 
entity that is of a technological character. The essence of technology is not itself 
technological (cf. Heidegger 1977: 4). That means that the essence of technology is 
not a thing; it itself is not a physical entity. Furthermore, he holds that the essence 
of technology is an activity: revealing or bringing something into unconcealment. 
His claim is that it is only as a basic process or activity that technology endures. 

The current discussion around AI is characterized by a similar tension. On 
the one hand side, intelligent technologies are conceived of simple means to ends. 
Processes in automation, robotics or speech recognition, to name only a few, are 
Al-based. These complex tasks require the ability to learn. Self-learning programs 
seem uncanny from the outside, but maybe not so much from the inside. Creators 
of such Als usually hold that there is not much intelligence hidden in the pro- 
grams. Rather it is a technological agency that reaches quite a level of sophistica- 
tion, but is far from being creative beyond the limits of its training. This task-ori- 
ented functional intelligence is to be sure continually evolving, but as of now only 
within certain limits and on the basis of the input the Al is trained with (cf. Pontin 
2018). 

Public discourse, on the other hand, is fueled by threatening scenarios of a 
singularity transcending human powers or, less futuristically put, threats of Al 
erasing jobs and manipulating human behavior (e.g., targeted personalized mar- 
keting). These issues arise from AI being generally opaque (ibid.), even if it is pos- 
sible to develop applications to observe AI learning processes (Sudmann 2018a). 
Also, the envisioned ubiquity of AI applications elicits broad discussions of the 
consequences for labor cultures (AI for optimizing work processes and automa- 
tion) and social environments (sensor-based observation systems). 

These preoccupations are related to Heidegger’s discussion of the essence of 
technology. What might be lying at the core of our preoccupations with Al is the 
fact that they are (or at least are envisaged) as world-making technologies. Technol- 
ogy according to Heidegger is not the sum of physical devices but above all a style 
of thinking and revealing entities. This aspect is made more and more explicit 
within the realm of future technologies. 

When we take a closer look at Heidegger’s words to describe the essence of 
technology, the relation to visibility and invisibility is undeniable. Describing the 
essence of technology as something that is itself not technological gestures toward 
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an invisibility. The transcendental conditions of technology are themselves not of 
a technological or objective character. Heidegger arrives at the idea that technol- 
ogy is essentially a way of world-making. The logic of enframing (Gestell) conceives 
of the world as standing-reserve (Bestand), i.e., a constellation of resources that is at 
disposal at all times. He finds this logic at work already long before modern tech- 
nology even emerged. While history tends to view modern physics as the enabler 
of modern technology, Heidegger holds that the structure or logic of technology 
already governs the development of modern physics (ibid.: 22 f.). The reason he 
gives for this claim is that modern physics as such is based on the belief that the 
world must be observable, measurable, and rendered predictable (ibid.: 172). Pre- 
dictability is necessary in order to treat the environment as standing-reserve. The 
interplay of needs and resources is a future- and hence prediction-based endeavor. 
Modern physics was already driven by the goal to tame the physical world through 
prediction and calculability, which is most explicit in the use of AI (e.g. for facial 
recognition used in border control or urban CCTV applications, and predictions 
of consumer behavior or optimizations of workflows through management Al). 
This means current usages of AI expand the potential to uncover standing-reserves 
beyond the exploitation of natural resources and thereby far into the depths of 
human behavior. The extent of this process is not yet clear, much less its conse- 
quences and ethical challenges. 

When technology constitutes the intelligibility of the world that reveals it as 
standing-reserve, as being always at disposal for our use, it also at the same time 
hides or conceals something. The way technology (or rather its essence, the pro- 
cess of enframing) insidiously compels humans to conceive the world as intelligible 
generally in terms of technology is tainted by the logic of instrumental thinking, 
of means and ends. It thus hides the character of objects as what stands over 
against subjects: “Whatever stands by in the sense of standing-reserve no longer 
stands over and against us as object.” (Heidegger 1977: 17) The process of revealing 
or making visible as described by Heidegger is perspectival, and a perspective also 
necessarily hides other or background aspects of the perceived objects. 

Visibility and invisibility condition each other in more than one aspect: In the 
case of technology, this interrelatedness or, to speak with Merleau-Ponty (1969), 
the chiasm (entanglement or intertwining) of visibility and invisibility goes deeper 
than in the case of perception. Perception is always situated and hence perspec- 
tival. There is no perception without a perspective. And that means there is no vis- 
ibility without the invisible. The dialectic of visibility and invisibility constitutes 
perception in general. 

Beyond the perception of technology as material objects/devices, which is an 
important topic in its own right (cf. Verbeek 2005), Heidegger sees a causality at 
work that is not exhausted by the instrumental definition of technology. Through 
technology we see the world as standing-reserve. Thus, technology produces visi- 
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bilities (the life-world as standing-reserve) rather than just adding (visible) objects 
to the world. As mentioned above these visibilities, or rather the all-encompassing 
style in which technology compels the world to appear as technological in general, 
also hides something, i.e., makes something invisible: namely, the objective char- 
acter of things as Gegenstände. This opens up another aspect within the broad topic 
of visibility. What is a thing when its thingness or Gegenständlichkeit is hidden? 

This is what happens when a tool like a hammer is used: The skilled user is not 
aware of the hammer as an object. Rather, the hammer becomes a prolongation 
of the body during usage. As long as the use remains frictionless, the hammer 
as object will not draw attention. It remains unthematic and its character as an 
object transparent. Such a use of things as tools is what Heidegger calls through- 
out his works “readiness-to-hand” or availability (Zuhandenheit): a description of a 
certain comportment toward things as being ready to use, being at our disposal. 
The instrumental attitude of technology makes things appear as means and hides 
their being as objects. 


3. Transparency and Opacity in Technological Objects 


If we translate this Heideggerian view of technology into a more common termi- 
nology, we arrive at a different form of visibility: namely, transparency. A tool or 
a technological device can be transparent in the sense that the user experience 
is smooth. Such a smooth user experience (or so-called “frictionless UX”) has be- 
come the gold standard in technology design and AI is one of the means to achieve 
this goal. A self-learning software can ideally learn from the user what it means 
to function smoothly. Any disruptions within the use of applications can further 
serve as materials from which it can learn and then create smoother functional 
processes that flow without disruptions. 

From a phenomenological perspective disruptions break the everyday attitude 
of smooth functioning and reveal the thingness and the character of objects as 
that which stands over against us (Gegen-stände). Only then will users have or find 
a reason to actually reflect on the technology. This also opens up the following 
possibility: In order to develop a critical attitude, disruption or friction is a nec- 
essary component. In neuroscience and philosophy of mind disruption or predic- 
tion errors is integrated in the model of neural activity as prediction processing: “In 
predictive coding schemes, sensory data are replaced by prediction error, because 
that is the only sensory information that has yet to be explained. (Feldmann & 
Friston 2010, p. 2).” (Cited by Clark 2015: 4) 

Conversely, this also reveals that functional transparency is at the same time 
associated with being opaque. The constitutive processes of a functioning technol- 
ogy and hence a smooth user experience has to stay hidden in order to perform 
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this job. In that sense, technological processes are supposed to be opaque: They 
remain hidden throughout the process of usage when they function smoothly. 
Transparency and opacity are manifest themselves like visibility and invisibility. 
The difference between the two pairs of concepts is that the case of visibility/in- 
visibility is a more neutral way to describe the givenness of objects in perception.’ 
Transparency and opacity tend to have a meaning that includes a normative aspect. 
At least this is the case when we broaden the perspective toward questions of de- 
mocratization or the potential of AI to foster democracy. 

To explain this train of thought in more detail, let me draw a line from the 
phenomenological use of the concept of transparency to its application in technol- 
ogy. Jean-Paul Sartre uses the concept of transparency in order to describe con- 
sciousness or, more narrowly, the imaginary, i.e., modes of consciousness related 
to images and phantasy. Consciousness constitutes perceptions in various modes 
without making the constitutive process itself perceptible. It remains transparent 
in its functionality, meaning that it does not become part of the object presented 
as perceived, remembered, or anticipated. By analogy, an AI application does not 
itself become an item of awareness when it functions smoothly. 

This becomes clear, when comparing different forms of givenness. For exam- 
ple, just now there is a cup of tea sitting on my desk. My act of seeing the cup 
of tea is an act of consciousness, an act of visual perception. This is one mode of 
how consciousness can present a thing: as given to vision, physically being there, 
within my reach. But the act of perceiving itself is not thematic, is not part of 
the intentional consciousness of the cup. The workings of consciousness remain 
transparent and they should do so, because otherwise something could be wrong 
with our eyesight or the overall state of health. If I remember the cup of tea later 
on, I will reproduce the visual characteristics of the cup through memory. The 
correlate of my memory is one produced by my imagination, which gives the cup 
to my consciousness as if I saw it. Again, the intentional act is perceived as an act 
of memory, but how this memory is constituted is not thematic in the memory 
itself. The workings of consciousness remain transparent. They are not thematic 


1 Edmund Husserl describes perception in his lectures on Thing and Space [1907] as being necessar- 
ily inadequate in the sense of necessarily involving aspects that are not directly perceived. Per- 
ception of a thing in space is always partial, being enriched step-by-step by changes in perspec- 
tive and the simultaneous quasi-perception (adumbration) of the hidden sides of the thing: “We 
see that the continuity of the corporeal thing presupposes ‘inadequate’ perception, perception 
through adumbrations that are always capable of enrichment and more precise determination.” 
(Husserl 1997: 101 [121]) This notion of perception necessarily includes perception of the non-per- 
ceived. That means human perception does not only conceive of things through adding perspec- 
tives consecutively to each other. Rather we are acquainted with spatial and temporal things in 
such a way that the hidden sides are perceived implicitly. This is what Husserl and Merleau-Ponty 
call “apperceptions”: The perception of the non-perceived. 
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in the process of cognition. In that sense, these processes are also opaque for the 
exploring mind. We have no conscious access to the inner workings of the mind. 
And this usually poses no problem. 

In the case of AI, however, it is different: Not knowing how an algorithm ar- 
rived at a solution can be highly problematic. If, for example, medical data are 
analyzed through an AI in order to identify a disposition for cancer, it is neces- 
sary to know on which grounds a diagnosis has been generated. Only on these 
grounds can a decision for preventive treatment be made. The problem is that an 
AI can generate predictions without being able to give a reason for the outcome, 
the choice of samples, or the method used. There is a categorical difference be- 
tween the causal processes leading to a mental state or an output of a program, 
and the ability to give reasons and reflect on mental states, as it is discussed with- 
in philosophy of mind. 

One can, for example, analyze the modes of consciousness through methods 
of phenomenological analysis and reflect on the different modes of intentionality 
in a given situation. Then consciousness as a process loses its transparency. The 
unthematic act of remembering or imagining becomes itself object of a higher or- 
der reflection. But then also a higher order of transparency emerges, namely the 
focus on constitutive processes of mental states becomes itself an object of per- 
ception and hence must itself be constituted. The infinite regress looms large here. 
The lesson to be learned from Edmund Husserl’s analysis of intentionality is that 
there is always a layer of consciousness that cannot itself be conscious because it 
itself constitutes a lower level or aspect of consciousness. Consciousness of tem- 
poral change, for example, cannot itself be temporal, at least not in the same way 
as the experience of time is: 


But we should seriously consider whether we must assume such an ultimate con- 
sciousness, which would be necessarily an ‘unconscious’ consciousness; that is to 
say, as ultimate intentionality it cannot be an object of attention [...], and therefore 
itcan never become conscious in this particular sense. (Husserl 2008: 394) 


Consciousness, therefore, is not only transparent as a medium of perception, it 
must also in some constitutive aspects remain opaque. We cannot understand 
consciousness simply by being conscious. 

Human consciousness is deeply influenced by technology and today in par- 
ticular by AI (cf. Hansen 2012, Hayles 2012, Stiegler 1998). The technogenesis of hu- 
man consciousness, as Katherine Hayles puts it, opens up another dimension of 
transparency/opacity. Al is a form of nonconscious cognition (cf. Hayles 2014) that 
becomes more and more ubiquitous. There is no online-shopping without sugges- 
tions generated by an Al; every social media news feed is individualized by algo- 
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rithms and even airfares are adapted to time, location, and devices. The virtual 
world is highly personalized through more or less sophisticated AI applications. 

Not only are the workings of the devices opaque in the sense that the user does 
not perceive the actual computational processes and even less so the data gather- 
ing that goes along with these processes. Even more so the output generated by Al 
applications does not necessarily reveal the underlying personalization processes. 
The Internet is only to a very limited extent a shared world. Most of the contents 
are shaped through user-Al interactions, though the user is not consciously aware 
of these interactions. Regarding technology in general, one can observe chang- 
es in human behavior and cognition with every new invention. The invention of 
writing, for example, has deeply altered how people memorize contents and how 
cultures preserve their traditions. The rise of smartphones has altered completely 
human ways of communicating. One simple example is communication through 
messaging devices and social networks: “tele-communication [...] entails a hid- 
denness of the face, a disappearance of the voice with its tonalities, the assum- 
ing of quasi-identities that do not authentically emanate from the concreteness 
of our being-in-the-world-in-the-flesh.” (El Bizri 2018: 130) One could find count- 
less examples of how new visibilities and at the same time opacities are generated 
through emerging technologies. 

The eerie twist comes with AI. Two factors are relevant: The temporal mi- 
croscale of computational processes and the predictive coding. The first factor, 
namely, the speed of computational processes that makes them inaccessible for 
human cognition, generates a scenario in which the second factor, namely, how 
the predictive coding turns into a preemptive force on human perception. As 
Mark Hansen writes in considering how computational processes that become a 
central element in the tissue of the life-world function on temporal microscales 
beyond our awareness: 


through the distribution of computation into the environment by means of now 
typical technologies including smart phones and RFID tags, space becomes anima- 
ted with some agency of its own. One crucial feature of this animation is its oc 
currence largely outside—or beside—the focal attention of actants within smart 
environments. For this reason, the intelligent space of contemporary life offers a 
kind of affordance—an unperceived or directly sensed affordance—that differs 
fundamentally from affordances as they have been theorized, following upon the 
work by James Gibson, in relation to media. When “we” act within such smart en- 
vironments, our action is coupled with computational agents whose action is not 
only (at least in part) beyond our control, but also largely beyond our awareness. 
(Hansen 2012: 33) 
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This description rests on the assumption that human cognition is constituted in 
relation with or by means of embeddedness in an environment. Hansen coins 
this as our “environmental condition” (ibid.), which describes the coupling of the 
individual and its environment. This coupling is not a static relationship, but a 
very dynamic one—a constant process of becoming. This refers to process on- 
tologies, which either hold that consciousness emerges from being embedded in 
an environment (cf. Merleau-Ponty 1969, Thompson 2013), or that consciousness 
even extends into the environment (a version of panpsychism, cf. Chalmers 2013, 
Whitehead 1929). Without delving into the environmental/ecology debate, I want 
now to transpose these thoughts into the context of smart environments and Al 
driven ecologies. 

Let me briefly summarize the train of thought leading up to this current 
juncture. I started out with Heidegger’s notion of the essence of technology as 
enabling condition of visibility or, more concretely, rendering the world percepti- 
ble as standing-reserve. This aspect of technology is itself not technological; rather 
it is the constitutive structure of technological thinking and thus underlies and 
makes possible the visible materiality of technological device. From there I took a 
detour into how human perception is constituted and showed that visual percep- 
tion is always situated and hence perspectival. That means aspects of invisibility 
are a constitutive part of vision or perception in general. The next step of my argu- 
mentation transposed the relation of visibility and invisibility into technological 
artifacts, where we speak of transparency and opacity, rather than of visibility 
and invisibility. Technological devices become transparent during use just as hu- 
man consciousness is transparent in perception (the process of the constitution of 
perception, for example, is not itself object of perception). Technological device 
function smoothly if there is no disruption and thus no reflection on process of 
usage required. This transparency is always accompanied by opacity. Although 
the mechanisms produce functionality, the computational processes remain hid- 
den, which is why digital technologies is often described as black boxes. This gets 
even more poignant with self-learning algorithms, which are not even fully un- 
derstood by their programmers. 

My aim is to show in the remaining sections how transparency/visibility and 
opacity/invisibility intertwine and establish new affordances. At this point I will 
go on with a reflection on smart environments. Smart environments or houses 
that are turned into an Internet of Things (IoT) exemplify a technology that is gov- 
erned by self-learning AI, whose main function is prediction. Prediction is neces- 
sary because the IoT within a household, for example, is a highly dynamic com- 
pound of interlinked processes that has to be adaptive for all kinds of situations 
and changes. Ultimately, I will argue that Al-driven smart environments differ 
strongly from low-tech environments for two reasons: (1) predictive responsive- 
ness has not been a common feature of environments before and (2) the prediction 
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and hence preemptive functionality is modelled around a conception of the ideal 
human/human behavior. It is here that the political discussion needs to start. 


4. Smart Environments: Technologies in the Tissue of the Life-World 


Intelligent technologies are being woven into the tissue or the flesh (Merleau-Pon- 
ty 1969, Rabari, Storper 2015, Förster 2018a) of the life-world, and it is important to 
understand that this is decidedly not a metaphor: Urban spaces consist of count- 
less sensors, cameras, and monitors. Especially megacities like Seoul, Tokyo, Lon- 
don, or New York City have CCTV in literally every corner of the city. Displays are 
present wherever you look and sensors measuring air quality, light intensity, or 
listening into the noise of the city go unnoticed, even if you start looking for them. 
The growing density of connected devices within smart environments creates a 
growing demand for very small hardware, integrated devices, and high-speed 
data nets. 

While urban spaces, work, and private spaces become more and more tech- 
nological, hardware in turn becomes less visible. Sensors see without being seen, 
and hear without being heard. This peculiar phenomenon makes up for the nar- 
ratives of future life-worlds, especially in contemporary science fiction movies. 
What is currently advertised or else emerging under the label of IoT or Internet of 
Everything (IoE) extends AI and thus nonconscious cognition into the last corners 
of the life-world. The topos of the vanishing of the hardware adds to the functional 
opacity of Al applications. Users have barely any chance to understand how Al is 
incorporated in devices when it is actually at work or how it shapes the process or 
experience of use. On top of this the physical implementation is no longer easy to 
locate. This means that technological environments are turned into a sensory, re- 
sponsive surface with nonconscious cognition. Dealing with responsive AI driven 
environments requires, therefore, new forms of knowledge and behavior, such as 
an understanding of technological agency. Nonconscious cognition and agency 
make up for fairly new affordances in daily life. On the one hand side, human 
behavior and movement needs to be adapted to the technological systems in or- 
der for them to work properly. On the other hand side, humans need to reflect on 
how they want these new technologies to be integrated in their life-worlds. This is 
precisely the point where an active engagement with new affordances and hence 
novel cultural structures needs to take place. 

The Al’s integrated in smart environments actively shape perception, move- 
ments, emotions, and rational choices (e.g., elections, ethical choices, etc.). One 
of the central problems is that Al’s exert their influence predominantly on the lev- 
el of affects (cf. Parisi 2018). This adds a third level of opacity or invisibility: the 
nudges generated by AI applications are not always perceivable as such. Recom- 
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mendations in shopping apps are quite straightforwardly nudges. The underlying 
structures of newsfeed generation are much less obvious. The way we retrieve in- 
formation from the Internet is always tainted through predictions of underlying 
learning algorithms. Thus, the world presented through a news feed is a person- 
alized world, generated by an AI that seems to know the user, while the user does 
not know how the program generates its output. The opacity of nonconscious cog- 
nition and agency, as it is operative in Al applications, creates uncertainties con- 
cerning current and future social life. Current science fiction movies are symp- 
tomatic for a more nervous human condition (cf. Férster 2016). There technology 
tends to be portrayed as a hidden force that goes through a cognitive evolution 
and eventually overpowers or leave humanity behind as an outdated life form (e.g. 
the movies Her (USA, 2013) and Transcendence [USA, 2014]). 

It is an undeniable fact that technology is becoming more and more invisible 
or at least smaller and more integrated within everyday objects and urban sur- 
faces. Even the skin as a limit is slowly breaking down. Sensors integrated in the 
body become increasingly more normal, even though the ethical dimension of this 
is debated. In Sweden, for example, some 3000 people already had such sensors 
implanted under their skin to replace keys, credit cards, or train tickets. There are 
two salient characteristics of distributed AI systems today: they become part of 
the environment (merging in tendency with everyday objects and surfaces, such 
as refrigerators, surveillance cameras, or Alexa voice assistant), or else parts of 
devices that function in close proximity to the body or become integrated within 
the body (clothes with smart fabrics, jewelry, or smart implants). One could say 
that technology becomes naturalized, if there ever was a clear-cut distinction be- 
tween the artificial and the natural to begin with. 

Smart environments are largely governed by AI because the sheer amount of 
data generated by the distributed net of devices needs to be digested and made 
useful. At this stage, we are faced with a complex structure of visibilities and their 
counterparts. Technology as hardware starts to disappear while its function-po- 
tential increases evermore exponentially. This tendency toward invisibility gener- 
ates a second- (or even third-)order transparency: Not only is technology in its us- 
age transparent, but it becomes transparent as an object. If technology had lost or 
hidden is object-character already according to Heidegger’s consideration, we are 
now reaching a higher level of enframing or Gestell. In Heidegger’s view technol- 
ogy obstructs our view of the world as object because it compels us to conceive of 
the world as standing-reserve. This basic characteristic also holds for technology: It 
does not appear as an object that stands over against a subject and, in this respect, 
transcends human aims. It is a means to an end that makes the environment ap- 
pear as a predictable, calculable reservoir of potentialities. This new layer of trans- 
parency comes into play through the disappearance of technological devices and 
the emergence of distributed AI. This makes a difference because most of what AI 
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today has accomplished is predictive and shapes functional processes according 
to those predictions. For the users, the unknown factors are huge: Users cannot 
know how exactly the system (e.g., IoT) or even one single device works. Moreover, 
they cannot know or actively experience better which, when, and how much data 
are gathered from the usage and behavior relating to these devices. Much less can 
be known of the use that is made of the collected data. The Big Data problem is be- 
coming discussed ever more widely, and the complexity grows with the increasing 
use of devices by the hour. 

From a phenomenological perspective, the decreasing visibility (and opacity) 
of smart technologies and their increased potential for agency is problematic for a 
democratization of AI. And the problem is not Al itself, whose actual intelligence 
is amazing but also constantly overrated. AI does not have the intention to build 
a better self, a better society, or a better future. Human beings aim for that. Phi- 
losophy is not a stranger to such mostly exaggerated goals. One of the obstacles 
to a transparent use of Al is this striving towards perfectibility, which is more or 
less an economic vehicle. Smart technologies have the potential to be useful and 
maybe even create a better future, but only if a culture of critique and open dis- 
course can be established and sustained. How does this point relate to the topic of 
visibility? Let me refer to Heidegger one last time. He argues that technology lets 
the world appear as standing-reserve. Today we should ask how human lives appear 
through technology. How do humans paint an image of human life by creating an 
environment that is machine-friendly? Do we have the means to make the hidden 
ratio of what it means to be human flourish in smart environments? How can we 
create enough freedom and potential for creative agency that allows for an ac- 
tive and critical engagement with existing technologies? That would imply exper- 
imenting, tampering, and first and foremost, conducting a critical discourse with 
industries relying heavily on predictions like retail and insurance businesses. The 
image of a “good” human life should be scrutinized (also with regard to the con- 
cept of the anthropocene). We need, therefore, a close observation of how nudges, 
prediction, and preemption influences everyday behavior—how we speak, move, 
and, indeed, smile or love. To do this successfully, humans in their whole range of 
diversity need to become visible and present as voices in public and in the indus- 
tries that rely heavily on AI. The political dimension of AI is very much a human 
one. The human image built into intelligent technologies needs to be made visible. 
Only then can an ethical discussion properly take place. 
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Introduction 


Any analysis of the intersection of democracy with AI must first and foremost en- 
gage the intersection of AI with pre-existing practices of marginalization. In the 
United States, perhaps no intersection is more salient than that of AI and race. 
As Al is increasingly positioned as the future of the economy, the military, state 
bureaucracy, communication and transportation across time and space, in short, 
as the bedrock of humanity’s future, questions of how Al intersects with pre-ex- 
isting practices of racial marginalization become central. These questions are par- 
ticularly difficult to answer given the black boxed nature of most contemporary Al 
systems. While it is certainly a worthwhile endeavor to push for increasing trans- 
parency into the datasets and algorithms powering Al systems, that transparency 
lies in an anticipated future and cannot help us now to analyze the operations of 
current AI systems. This picture is only complicated by the fact that AI systems, 
particularly those operating at web scale, are difficult for even their engineers to 
understand at later stages in their operation. For instance, a programmer may be 
able to easily describe the seed data and the machine learning algorithm that she 
started with, but may be completely unable to explain the rationale behind the 
subsequent classifications that the system learns to make. Again, it is certainly 
worthwhile to call for AI explicability—namely, requiring AI programmers and 
engineers to develop systems that can explain their decision-making processes or, 
in the most extreme case, only make decisions that can be explained clearly to a human— 
this again is an anticipated future that is of little use to answering the immediate 
question of race and AI which already has dire consequences at this very moment. 

So how can an outsider go about critically analyzing the intersection of race 
and Al in the contemporary moment? This paper will utilize an interdisciplinary 
methodology that I am calling ‘speculative code studies’, which combines archi- 
val research into press releases, company blog posts, science and technology jour- 
nalism, and reported instances of technological irregularities with critical code 
studies research into the available datasets that machine learning algorithms are 
trained on, analyses of open-sourced variants of black boxed code, and empirical 
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studies of the outputs of black boxed systems. The goal of such a study is to pro- 
duce a rigorous, but speculative, analysis of black boxed code. This analysis must 
always remain speculative, as the actual systems are obfuscated from direct anal- 
ysis, but the methodology ensures that the analysis is as rigorous as possible given 
the peripheral materials, segments of code, and inputs/outputs that are available. 
In this chapter I use this method to probe the myriad ways in which racial biases 
that are present both in the boardrooms and research and development wings of 
technology companies and in the broader socio-cultural milieu get hardcoded into 
the datasets and, subsequently, the machine learning algorithms built atop them. 
It will be my argument that many of these algorithms constitute a material man- 
ifestation of racial bias. 

This paper will primarily be concerned with visual or optical media, and com- 
puter vision algorithms in particular. I will argue that within this context black- 
ness falls into the notorious dialectic of hypervisibility and invisibility—black- 
ness is too often rendered in stereotypes, at times even visually cartoonish, or it is 
rendered as systemically invisible. However, it is important to note that while the 
context of my analysis is systemic racism against predominantly African Amer- 
icans, these systems have global impacts for people of color—or, to speak more 
precisely, those darker-skinned individuals who fall within types V or VI of the 
Fitzpatrick scale (cf. Fitzpatrick 1975, 1988). When I use terms like ‘black,’ ‘person 
of color, or ‘dark skin,’ it is meant to indicate that the problems I am identifying 
are of global concern and have high stakes impact on people at the darker end of 
the Fitzpatrick scale across the planet, even though my analysis is contextualized 
within the history and culture of the United States. It is outside the purview of this 
chapter to extend this analysis to other conjunctures, but I sincerely hope others 
will help me to do so by extending, revising, and challenging this work. In the 
first section of this chapter I will draw on critical race theory to demonstrate how 
this dialectic is problematic from the perspective of egalitarian democracy. In the 
next section I will offer a brief overview of the history of racial bias in visual media 
within the context of the United States that perfectly illustrates this hypervisi- 
ble/invisible dialectic of blackness. In the following two sections I will look at the 
hypervisibility and invisibility of blackness in contemporary AI systems and will 
try to demonstrate the enormity of the stakes of this conjuncture. In conclusion I 
offer some preliminary thoughts about where we can go from here. 
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The Hypervisibility and Invisibility Dialectic of Racial Difference 


“Black is ... ‘an black aint.” 
Ellison 1989, p. 9 


In their theory of racial formations, Michael Omi and Howard Winant (2015) have 
argued that humans essentially use stereotypes to make sense of the world, even 
though these stereotypes are constantly changing. People make use of fundamen- 
tal categories of difference, like race, gender, class, age, nationality, and culture 
to navigate society, all of which imply a certain politics of “othering” that pro- 
duces structural marginalization, inequality, exploitation, and oppression. We 
might productively understand machine learning as engaging in a very similar 
behavior with similar political stakes. As I have argued elsewhere in the context 
of machine learned semantic labels, these algorithms engage in an iterative pro- 
cess of learning stereotypical differentiations to categorize the various data that 
they encounter (Monea 2016, 2019). However, race functions differently than these 
other stereotypical categorizations because, as Omi and Winant explain, race is 
crucially ‘corporeal’ and ‘ocular’. What they mean by this is that racial distinc- 
tions take hold of a set of phenotypic differences—most noticeably morphologi- 
cal differences like skin tone, lip size, and hair texture in the case of blackness in 
the United States—and essentializes them, as if they were physical markers of an 
essential difference of kind (2015: p. 13). It is thus othering, as it establishes the 
border between an ‘us’ and a ‘them, and reifies that border by making it appear as 
a fundamental law of nature, a scientific fact, a marker of a different kind of being. 
There are two unique aspects of this process, which Omi and Winant refer to as 
‘racialization’. First, these phenotypic differences are arbitrarily selected, are not 
understood as having the same denotations and connotations across space and 
time, and often were previously unconnected to any racial classification. Second, 
they are written on to the body through morphological distinctions in such a way 
that racial difference is legible on sight alone. 

This latter aspect of racialization has been a core component of critical race 
theory for decades, and was perhaps most notably articulated by Frantz Fanon 
(1967) in his concept of ‘epidermalization’. For Fanon, epidermalization is a pro- 
cess by which black people realize their identification as the Other for white peo- 
ple as they encounter the white gaze that dissects and analyzes their body, without 
permission, to classify them. Fanon writes: 


| am overdetermined from the outside. | am a slave not to the “idea” others have of 
me, but to my appearance. [...] The white gaze, the only valid one, is already dissec- 
ting me. | am fixed. Once their microtomes are sharpened, the Whites objectively 
cut sections of my reality. (1967: p. 95) 
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Stuart Hall succinctly defines Fanon’s idea of epidermalization as “literally the in- 
scription of race on the skin” (1996: p. 16). Hortense Spillers similarly writes about 
the ‘hieroglyphics of the flesh’, wherein black subjects are transformed into flesh 
through “the calculated work of iron, whips, chains, knives, the canine patrol, the 
bullet” (2003: p. 207). For Spillers, Western humanism is built atop these hiero- 
glyphics, as the liberated Man requires definitionally that an other be designated 
as not fully human. This legacy is passed down through the generations even after 
black subjects were granted possession of their own bodies and continues to struc- 
ture our social lives. As Alexander Weheliye describes it, racial categories “carve 
from the swamps of slavery and colonialism the very flesh and bones of modern 
Man” (2014: p. 30). Sylvia Wynter (2001) has similarly shown how this happens in 
her arguments about sociogeny, where a focus on phenotypical differences is just 
a ruse to essentialize racial difference and divide the Homo sapiens species into 
humans and nonhuman beings. 

We can understand this corporeality, ocularity, epidermalization, and flesh- 
iness of race as a fundamentally visual component, and one that makes race hy- 
pervisible by stressing phenotypes—especially morphological features like skin 
tone, lip size, and hair texture—and connoting racial stereotypes that help bolster 
racial marginalization. Lisa Nakamura (2007) has shown how this hypervisibility 
of race is perpetuated today by computation and digital visual culture, leading to 
the production of ‘digital racial formations’. John Cheney-Lippold (2017) has sim- 
ilarly shown how algorithms have digitized race into ‘measurable types’, or sta- 
tistical probabilities based on user data. Here I’d like to introduce the term ‘users 
of color’ to replace the term ‘people of color’ for this digital context. Users of color 
are digitally racialized based on algorithmic analysis of big data, which is reifying 
some old phenotypic stereotypes of racial difference at the same time that it is 
producing new ones. We might also follow Simone Browne (2015) and think of this 
process as a ‘digital epidermalization’. 

Now, we can easily see that even for technology companies that may have lit- 
tle ethical commitment to egalitarian democracy, the public relations nightmare 
alone of being seen as reifying racial stereotypes in digital culture is a huge deter- 
rent. As well see throughout this piece, the most frequent response to criticism of 
algorithmic racialization is to make race invisible. Rendering blackness invisible 
is always the flipside of the coin, in dialectical tension with racial hypervisibil- 
ity. Both options are unsatisfactory, as they both provide safe haven for racism, 
albeit in different ways. The most recent and visible example of rendering race 
invisible is the ‘color blind’ policies that have been pursued in the United States 
since the 1960s. These policies have been near universally condemned by scholars 
in critical race studies and related disciplines (e.g. Bonilla-Silva 2017; Brown et 
al. 2003; Omi/Winant 2015). Color blindness delegitimates affirmative action and 
similar programs striving for racial equality, allows racism to operate unchecked 
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provided it uses dog whistles and other careful language to obscure racial malice, 
and obscures important racial data trails that might otherwise have been used to 
uncover statistical trends of racism (e.g. in policing, court sentencing, allocation 
of welfare benefits, etc.). In addition, it has made speaking about race so taboo 
that the term ‘white fragility’ was coined to describe contemporary white people’s 
inability to openly talk about race and racism (Dyson 2018). Thus, rather than deal 
with racialization’s roots in colonialism and slavery and doing the hard work of 
moving towards actual egalitarian democracy, blackness is alternately rendered 
as hypervisible or invisible, both of which leave much to be desired. In short, as 
Ralph Ellison put it, “Black is ... ‘an black ain’t” (1989, p. 9). 


Accounting for the Visibility of Race in Visual Media 


“Photography is a weapon” 
—Oliver Chanarin (PhotoQ 2015) 


The United States has a long history of embedding racial stereotypes in its visu- 
al media and communications technologies (cf. Dyer 1997, hooks 1992, Nakamura 
2007). This legacy spans from analog to digital photography and, as we'll see, con- 
tinues to impact a number of computer vision applications. For example, Lorna 
Roth (2009) has shown in detail that Kodak optimized its entire suite of prod- 
ucts for white skin. Kodak produced a long series of “Shirley cards”, named after 
Shirley Page, the first studio model for the photos Kodak sent out with its new 
products. These Shirley cards were marked “normal” and used as test cards for 
color balancing film stock and printers. The optimization for white skin was to 
the detriment of people of color, whose features increasingly disappeared in di- 
rect correlation with how dark their skin was. One legendary result was French 
film auteur Jean-Luc Godard’s refusal to shoot on Kodak film for an assignment in 
Mozambique because of the racial bias hardcoded into the film—it literally would 
not work in Africa. It was not until complaints were made by companies trying to 
photograph dark objects for advertisements that Kodak developed film that could 
capture the details of black flesh, a project that was kept quiet at Kodak and re- 
ferred to via the coded phrase “To Photograph the Details of a Dark Horse in Low 
Light” (Broomberg & Chanarin n.d.). 

Even when film was made with the explicit intent of rendering black skin visible— 
rather than for rendering chocolate bars and wooden furniture visible in advertise- 
ments—deeply rooted racial problems cropped up. Take, for instance, Eric Morgan’s 
(2006) story of the Polaroid ID-2 camera. The ID-2 was designed to take two photos 
per self-developing print, one portrait and one profile image, of a subject 1.2m from 
the lens. This was, in essence, a streamlining of Alphonse Bertillon’s anthropomet- 
ric identification system for state policing, which has always intersected in complex 
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ways with existing practices of racial and gendered marginalization (Browne 2015; 
Fair 2017; Lyon 2008; Wevers 2018). In the case of the ID-2, this was manifested in a 
special “boost” button for the flash that would make it around 40 percent brighter, 
the same amount of light that darker skin absorbs. In and of itself, this feature is 
rather innocuous. It is actually a step forward in the sense that it allowed the camera 
to capture the features of dark-skinned people more clearly, although still made it 
problematic to capture both black and white skin in the same picture. However, in 
1970, a Polaroid chemist named Caroline Hunter uncovered evidence that Polaroid 
was making a lot of money selling ID-2 cameras to the South African government, 
which used them to make the passbook’s that black citizens were forced to carry with 
them at all times within white areas (see Savage 1986). 

In their exhibit titled after Kodak’s coded phrase “To Photograph the Details 
of a Dark Horse in Low Light,” South African artists Adam Broomberg and Alex 
Chanarin argue that the ID-2 was designed for the purpose of supporting apart- 
heid (PhotoQ 2015; Smith 2013). It is unlikely that this is the case, but it is cer- 
tainly true that a certain number of Polaroid executives had a damning amount 
of knowledge of the trafficking of ID-2s in South Africa and worked to intervene 
too late and with too little energy to actually prevent the use of their new technol- 
ogies for the support of Apartheid. Neither of these stories is meant to minimize 
the technical difficulties of capturing darker features and objects on film. These 
difficulties are inherent to optical media and would likely exist no matter the so- 
ciocultural context within which photographic technologies arose. What they do 
expose, however, is how absent and unimportant the dialectic of black invisibil- 
ity and hypervisibility was to these companies, a fact that was made materially 
manifest in their research and development paradigms, the products they took to 
market, the discourse they used to position these products, and their responses 
to criticism from the public. It is not that these devices themselves are racist, but 
instead that racial biases from the context in which they are developed inflect how 
the companies approach research and development, imagine certain products 
and not others, prioritize some more highly than others, assess some potential 
bad applications of their technology as negative press coverage to be accounted for 
in the design process, etc. All of this is materially manifest in the technology itself, 
the way it is positioned discursively, and the myriad uses it gets put to in society. 

It wasn’t until the ‘90s that American companies really began to take seriously 
the need to better capture black features on film. Kodak released ‘black’, ‘Asian’, and 
‘Latina’ Shirley cards and began marketing Kodak Gold Max film, which had an ‘im- 
proved dynamic range’—it could finally ‘photograph the details of a dark horse in low 
light’. These achievements were made by new innovations in the chemical composi- 
tion of film negatives to make them more reactive with darker pigments and modi- 


1 For more on Broomberg and Chanarin’s exhibit, see O’toole 2014. 
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fications to color balancing techniques to optimize for different skin tones. They all 
required an increasing focus on the visibility of people of color that spanned from 
the consumer base to the research and development teams at companies like Kodak. 
This change in priorities also quickly manifested in digital camera and camcorder 
technologies. For example, in 1994 the U.S. Philips Corporation filed for a patent on 
a new ‘tint detection circuit’ that could automatically adjust the tint in a digital im- 
age to white balance for both light and dark skin tones at once. In this instance (see 
Figure 1 below), there are literally two components soldered onto the board, one for 
‘SKINR and one for ‘SKINB’ that lead to an ‘AND CIRCUIT’ that combines their out- 
puts to optimize for both. While this solution is not ideal, as the color balancing will 
be off for both light-skinned and dark-skinned people if they are in the same shot, it 
is a step in the right direction. It will work equally well for any skin type in isolation 
and when in the same frame will average towards the middle, rendering both poles 
of the Fitzpatrick scale equally less visible. We might also take this historical arc as 
emblematic of the level of intervention necessary if we are to make any given tech- 
nology democratic as it pertains to racial marginalization. Just as we need to literally 
hardcode anti-racism measures into the circuit board to get an egalitarian camera, 
we will need to literally hardcode anti-racism measures into our machine learning 
algorithms to get egalitarian Al. 


AUTOMATIC 
WHITE BALANCE 
CONTROL CIRCUIT 


Fig. 1: U.S. Patent No. 5,428,402 (1995) 
The Hypervisibility of Race in Al Systems 
In a ‘humorous’ story read for This American Life, David Sedaris describes the 


strange cultural differences surrounding Christmas between the United States 
and the Netherlands, noting to refined laughter that in the Netherlands, Saint 
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Nicholas is accompanied by six to eight black men who prior to the mid-1950s were 
characterized as his personal slaves (Sedaris 2001; cf. Sedaris 2004). In Dutch tra- 
dition, these black helpers eventually stabilized into the image of Zwarte Piet, or 
‘Black Pete’, with Dutch men and women dressing up in black face—black face 
paint, red lipstick, curly black-haired wigs, a golden hoop ear ring, and colorful 
Spanish/Moorish outfits—to lead Sinterklaas, or Saint Nicholas, in parades, dis- 
tributing candy and kicks to good and bad children respectively. The majority of 
Dutch citizens have very positive attitudes about Zwarte Piet, and often downplay 
the connections between him and black face by noting alternately either that his 
face is colored such because he is a chimney sweep that now crawls through chim- 
neys to deliver candies to good children who leave their shoes out or because he is 
a Moor that was adopted by Sinterklaas, who lives in Spain in the offseason rather 
than at the North Pole, as in American traditions. As Allison Blakely notes, both of 
these explanations are rather unconvincing (2001: pp. 47-48). 

This desperate attempt to preserve a deeply problematic tradition are only com- 
plicated by the fact that Zwarte Piet is the name for the Devil in Dutch folklore, who 
is caught and chained for the celebration every year, and by the Netherland’s compli- 
cated history with colonialism, the slave trade, and slavery—for example, the term 
‘apartheid’ comes from the Dutch and arose particularly in the context of their colo- 
nial occupation of Surinam (Blakely 1980: p. 27). While the Dutch position themselves 
as a nation apart from colonial and racial marginalization, this has never been the 
case, especially since a number of Surinamers relocated to the Netherlands after the 
1950s and have faced systemic marginalization based on race (Blakely 1980). It is no 
wonder that these same Surinamers are increasingly unenthused with the Zwarte 
Piet tradition and argue that it is insulting, and especially damaging to children of 
color who subsequently face bullying at school (Blakely 2001: p. 48). 

David Leonard has argued that a more appropriate metric for determining 
culpability in instances of blackface is not whether the person dressing in black- 
face meant to offend people, but whether that person is causing harm, either to 
individuals or to society (Desmond-Harris 2014). This is a much smarter way of 
analyzing the situation, as it not only rids us of complex interrogations of inten- 
tionality, but also opens up the analysis of how photographic or videographic me- 
dia might extend the range of impact of that harm. For instance, barring for the 
moment the issue of blackface in the Netherlands, it is certainly the case that it is 
extremely damaging in the context of the United States. As C. Vann Woodward 
(2001) has shown the campaign of pseudo-scientifically legitimated racism, dehu- 
manization, segregation, disenfranchisement, and terror waged against African 
Americans that we know as “Jim Crow” was named after a blackface minstrel rou- 
tine. Blackface thus encapsulates rather neatly the logic of American racism, in 
that it literally denotes and makes hypervisible phenotypes—and in still images 
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these are primarily morphological traits—while at the same time always preserv- 
ing their connotation of inferiority in connection with white supremacy. 

If we look at the case of ImageNet, we can clearly see how these Zwarte Piet imag- 
es escape their context and cause social harm. ImageNet is a large dataset of labeled 
images first launched out of Princeton University in 2009. The dataset originally 
drew 80,000 labels from the semantic database WordNet—each label is referred to 
as a ‘synset’ which is a set of synonymous terms—with the goal of populating each 
of these labels with 500-1,000 clean and full resolution images (Deng et al. 2009). 
In 2010, ImageNet launched its annual ImageNet Large Scale Visual Recognition 
Challenge (ILSVRC), which has since served as an industry benchmark for success in 
computer vision applications (Russakovsky et al. 2015). This centrality was cement- 
ed by Alex Krizhevsky, Ilya Sutskever and Geoffrey Hinton’s (2012) groundbreaking 
success in using neural networks to win the competition to produce an algorithm 
that could learn to classify images in the ILSVRC (cf. Sudmann 2016, 2018). What is 
important to know is that ImageNet not only serves as the performance benchmark 
for nearly all computer vision systems, but that because those systems are trained on 
and optimized for the ImageNet dataset, any biases in the ImageNet dataset have 
wide-ranging repercussions since they subsequently become hardcoded through 
machine learning into a large portion of computer vision systems. 


Fig. 2: Zwarte Piets and Sinterklaas (Splinter 20107) 


2 N.b. | could not establish the licensing for any of the Zwarte Piet images contained in ImageNet, 
but any search on Flickr for the term returns images like this one which are illustrative. 
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ImageNet gathers images for a synset that contains the terms “Black”, “Black per- 
son”, “blackamoor”, “Negro”, and “Negroid”, which it defines as “a person with dark 
skin who comes from Africa (or whose ancestors came from Africa)”. This synset 
is interesting for a number of reasons. First, just from browsing it, one can tell 
that it contains many fewer images as a percentage of the total images than other 
synsets that would contain useful visual details for building out a classifier. What 
I mean by this is that an inordinate number of photos are low resolution, don’t 
show facial details, have black people’s bodies positioned further away from the 
camera, and inordinately feature celebrities (about 1 per cent of the entire dataset 
is pictures of Barack Obama) and memes. While this certainly isn’t a smoking gun 
for racial bias, let alone intentional racial bias, it does reveal that the capacity to 
accurately identify black facial features is not prioritized by default for any com- 
puter vision algorithms trained on ImageNet’s data. What is perhaps closer to a 
smoking gun is that of the 1,286 images for this synset that are still available on- 
line, a full 79 of them are of people in blackface.’ All but one of these images are 
of people dressed as Zwarte Piet. Thus, this odd Dutch phenomenon has the exact 
consequences that are feared in critiques of blackface: regardless of the intentions 
of those wearing blackface, when it enters public discourse, this signifier of black- 
ness quickly escapes any contextualization and instead reifies racism through its 
connotations of white supremacy and its denotations of blackness being reducible 
to phenotypical difference. We can see this quite literally in the case of ImageNet, 
where blackface images have escaped their context to compose just over 6 per cent 
of the entire dataset, a dataset developed on a continent where the vast majority 
have never even heard of Zwarte Piet. Even if we allow the extremely dubious ar- 
gument that these images are harmless in the Netherlands, a tradition beloved by 
fewer than 20 million has helped to bias a fundamental piece of infrastructure for 
computer vision. 

Take, for example, Google’s use of the ImageNet dataset. In 2014, Google re- 
searchers won the ILSVRC challenge with their ‘Inception’ algorithm (also known 
as ‘GoogLeNet’), a 22-layered convolutional neural network (Syzegedy et al. 2015). 
This CNN was trained on the ImageNet dataset, and thus internalized any biases 
present in that data.* At Google I/O 2015, Google announced the launch of its new 


3 Atthis time, | can only work from the publicly accessible list of links to images for given synsets. 
This means that | cannot access images that have since been taken down from the web, which 
makes it impossible to access the full set of 1,404 images that make up the dataset for that synset. 
| have repeatedly submitted requests to register for an account with ImageNet so that | might 
access the full dataset and have sent multiple emails to ImageNet’s contact address, but have yet 
to get either access or a response. 

4 While there is no comparable data for racial biases, it has been demonstrated that in the case of 
gender biases, neural networks not only internalize the biases in their datasets, but amplify them 
(Zhao et al. 2017). 
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Google Photos software. Google argued that humans were now taking over a tril- 
lion images a year and at this rate would need a second lifetime to label, organize, 
and revisit their photos. Google Photos was the solution we had all been waiting 
for: 


Google Photos automatically organizes your memories by the people, places, and 
things that matter. You don’t have to tag or label any of them, and you don’t need 
to laboriously create albums. When you want to find a particular shot, with a simp- 
le search you can instantly find any photo—whether it’s your dog, your daughter's 
birthday party, or your favorite beach in Santa Barbara. And all of this auto-grou- 
ping is private, for your eyes only. (Google 2015) 


The new Google Photos software was primarily powered by the Inception/Goo- 
gLeNet algorithm that was trained on ImageNet data, though it supplemented 
the image patterns it learned from ImageNet with a huge database of photos and 
nearby text from websites it had crawled and a few other indicators, like looking 
at the place and time stamps of both the user during the search and the images 
via their metadata (Brewster 2015). The centrality of ImageNet is no wonder, as it 
not only serves as the benchmark for computer vision algorithms and is standard 
across a large portion of the industry, but Google Photos was developed under 
Bradley Horowitz, Google’s Vice President of Streams, Photos, and Sharing, who 
previously saw the value in Flickr’s Creative Commons licensed images when he 
helped purchase the company as an executive at Yahoo (Levy 2015). 

The point to be taken from all this is that Google Photos was designed by peo- 
ple who viewed ImageNet as an unquestioned industry standard and who placed 
strong faith in the utility of Flickr images. They thus were ill positioned to fore- 
see the racial biases inherent in the visual data that their machine learning al- 
gorithms had used to develop their classifiers. This problem came to the fore just 
a month after Google Photos was released when in June 2015 a black software 
engineer named Jacky Alcine posted a set of images run through Google’s photo 
tagging software to Twitter in which images of him and his girlfriend were labeled 
as photos of ‘gorillas’ (Alcine 2015).° The case clearly hit a nerve, as dozens of arti- 
cles were published calling Google’s algorithms racist within days and it has since 
been one of the most frequently cited examples of algorithmic bias in technology 
journalism. This comparison is only possible based on an arbitrary definition of 
certain phenotypical differences as the sole markers of an essentialized differ- 


5 Itis worth noting that while Google is the highest profile instance of this happening, it is by no 
means the sole incident. Just a month prior to the Alcine incident Flickr made news for mislabel- 
ing a black man (and a white woman) as “animal” and “ape”, and also labeled photos of Dachau 
concentration camp as “jungle gym”, “sport”, and “trellis” (Hern 2015). 
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ence. In short, when we think of skin tone, lips, and hair as the cornerstone of ra- 
cial difference, this slippage between classifiers is opened up. And further, when 
we essentialize these differences and connect them to racist stereotypes, the con- 
notation of this slippage becomes unbearable. It calls into question not only peo- 
ple’s intelligence, but their very humanity. This is a problem of the highest stakes 
for all users of color. 

As can be seen from Google’s response, the company similarly understood 
this instance to be a serious problem and one that might threaten the future of 
their computer vision platforms. Within hours, a Google engineer named Yonatan 
Zunger was responding to Alcine’s tweet asking for permission to examine his 
Google Photos account to figure out what had gone wrong. The next day, Zunger 
tweeted that Google had not recognized a face in the images of Alcine and his 
girlfriend at all and noted, “We're also working on longer-term fixes around both 
linguistics (words to be careful about in photos of people [lang-dependent]” and 
in “image recognition itself. (e.g., better recognition of dark-skinned faces) (Zu- 
nger 2015). Zunger promised that Google would continue to work on these issues, 
which included developing systems that could better process the different con- 
trasts for different skin tones and lighting. A few days later a Google spokesperson 
told the BBC that, “We're appalled and genuinely sorry that this happened. We 
are taking immediate action to prevent this type of result from appearing” (BBC 
2015). Yet, as we'll see in the next section, Google has yet to discover a solution for 
this problem. The datasets its algorithms are trained on make it so that race must 
be rendered either as a hypervisible emphasis on phenotypes, which, without a 
heavily curated new dataset will continue to produce slippage between users of 
color and apes, or as invisible. 


The Invisibility of Race in Al Systems 


While the speed and sincerity of Google’s initial response seemed promising, af- 
ter more than two years WIRED reported that all Google had managed to do was 
remove potentially offensive auto-tags for terms from its Photos software (Si- 
monite 2018). In that same report, WIRED noted the results of a series of exper- 
iments they had done with Google Photos. First, they ran a collection of 40,000 
images well-stocked with animals through the system and found that it did not 
locate any results for the terms “gorilla”, “chimp”, “chimpanzee”, and “monkey”. 
In a second experiment they tried uploading pictures solely of chimpanzees and 
gorillas and found that it still would not recognize the offending set of terms. In 
a third and more damning test, WIRED uploaded a collection of over 10,000 im- 
ages used for facial-recognition research. Searching these photos for auto-tags 


of “black man”, “black woman”, or “black person” only delivered photos that were 
in black-and-white, which did correspond to the gender specified, but did not sort 
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people by race. In short, in response to this public relations disaster, blackness has 
become invisible in Google Photos.‘ This color blindness also extends to Google’s 
Open Images dataset, which contains “30.1M image-level labels for 19.8k concepts, 
15.4M bounding boxes for 600 object classes, and 375k visual relationship anno- 
tations involving 57 classes” (Kuznetsova et a. 2018). None of these millions of la- 
bels, thousands of concepts, or hundreds of classes, from what I can gather after 
examining the database, explicitly label race. Users of color are only identified by 
their absence. 

The issue of black invisibility has a long history in systems that process visual 
data for applications like facial recognition and motion-sensing. Take, for exam- 
ple, the viral 2009 YouTube video of an HP laptop designed to use facial recog- 
nition to track users’ faces and follow them as they move with the webcam that 
failed to register the movements of ‘Black Desi’ at all, despite easily following the 
motions of his white coworker (wzameno1 2012). Consumer Reports (2009) tried to 
debunk the argument that this was a racial bias by arguing that it instead is a 
factor of lighting conditions, and while they present their results as if the system 
would work the same for lighter and darker skin tones in the same lighting, in their 
video it is clear that this is not the case, as they have to better light their user of 
color’s face before the system starts to track him. For another example, take Xbox 
Kinect, which in 2010 was reported to have trouble recognizing the faces of users 
of color (Ionescu 2010). This primarily effected their ability to automatically log in 
to their avatars, as Kinect gameplay largely functions on skeletal movement. In 
other words, Kinect is capable of seeing black bodies but not black faces, and can 
facilitate their gameplay so long as it doesn’t need to recognize their face, which 
some games do. Consumer Reports (2010) similarly argued that this was merely a 
lighting issue and claimed to have ‘debunked’ the idea that Kinect is ‘racist’. 

It is certainly the case that these machines themselves are not intentional 
agents engaged in prejudicial thinking, but to merely wave away the claims of 
racism as ‘debunked’ after demonstrating that failure to function appropriately 
when utilized by people of color requires an uncomfortable amount of hubris and 
near total lack of empathy. There is something clearly going on here and it has a 
felt impact on racialized bodies so clear in the videos of people attempting and 
failing to utilize facial recognition and motion-sensing technologies. At the very 
least, these instances are material embodiments of some combination of the lack 
of forethought in the research and development phase and a lack of concern over 


6 Interestingly, this invisibility plays out differently in Google’s less publicly visible computer vision 
projects. Google’s Cloud Vision API launched in 2016 as a new tool to make its computer vision 
algorithms available particularly to developers (Google 2016). Cloud Vision API still uses labels 
like “chimpanzee” and “gorilla”, though I’ve found no studies to date of whether the same racial 
hypervisibility of Google Photos persists on Google’s Cloud Vision API. 
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going to market with a product that would fail to operate for a large and protected 
minority class of citizens. And further, the common arguments that all technolo- 
gies fail, that these technical constraints are unavoidable, and that it is common 
sense to design products based on your majority market (and in most instances 
this is code for ‘white’ people) are all inadequate at best, and deeply offensive at 
worst. 

Take, for another example, the frequent instances of motion-activated devices 
like soap dispensers in public restrooms failing to recognize users of color (e.g. 
Fussell; Plenke 2015). Bharat Vasan, the COO of Basis Science, explained to CNET 
that there are systems that can avoid this problem by detecting the darkness of ob- 
jects beneath their sensors and adjusting a spotlight to match, but these systems 
are too expensive for many of these lower-end motion-activated devices (Profis 
2014). It is here that chronic lack of consideration for users of color becomes most 
apparent. These systems are designed for public use, and thus by default require 
consideration of users of color. Further, many aspects of the lighting conditions 
can be predicted in advance (e.g. fluorescent overhead lighting, often with the 
shadow of the motion-sensing object itself falling over the object to be detected). 
This picture only gets more complicated when we consider that the principle site 
of research and development for automated restroom innovation is in prisons, 
which are disproportionately comprised of people of color in the United States 
(Edwards 2015). 

This is a much more significant problem than simply having automated public 
utilities that fail to operate for users of color. These same problems extend to med- 
ical technologies and limit the effectiveness of new wearable technologies tout- 
ed as breakthrough technologies for everyday medical monitoring. For instance, 
Pulse Oximetry, which optically measures arterial hemoglobin oxygen saturation 
is demonstrably less effective in people with darker skin tones (Bickler/Feiner/ 
Severinghaus 2005). While the FDA requires these devices to meet certain accu- 
racy thresholds before they can go to market, they do not specify where on the 
Fitzpatrick scale the test subjects must fall. There is thus a financial incentive to 
use lighter-skinned test subjects, as utilizing and designing for darker skin types 
slows their path to market. New non-invasive neuroimaging techniques like func- 
tional near-infrared spectroscopy (fNIRS) are being used to study and potentially 
treat medical issues like Alzheimer’s disease, Parkinson’s disease, epilepsy, trau- 
matic brain injury, schizophrenia, mood disorders, and anxiety disorders (Irani et 
al. 2007). {NIRS measures brain activity through blood oxygenation and volume 
in the pre-frontal cortext and is similarly less effective with darker pigmentation 
and darker, thicker hair (Saikia/Besio/Mankodiya 2019). These same problems of 
black invisibility extend to the optical heart trackers installed in many contempo- 
rary wearable technologies, like FitBit and the Apple Watch (Kim 2017; Profis 2014). 
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Perhaps the ultimate example of the algorithmic invisibility of users of col- 
or though can be found in lidar-based state-of-the-art object detection systems 
like those used in autonomous vehicles or ‘self-driving cars’. Autonomous vehicles 
have to engage in grisly cost-benefit calculations in crash scenarios to determine 
how to kill or injure the fewest people (e.g. Roff 2018). New research has come 
out demonstrating that such systems are statistically less likely to identify dark- 
er-skinned pedestrians as humans to be factored into these calculations (Wilson/ 
Hoffman/Morgenstern 2019). The researchers found that this remained true even 
when you factor in time of day (i.e., lighting, the go-to excuse for any technolog- 
ical failure to recognize black faces and bodies) and visual occlusion. This could 
have been predicted, as the training data used for the system they analyzed con- 
tained nearly 3.5 times as many images of light-skinned people as dark-skinned 
people (ibid.: p. 1). Black invisibility is thus not merely a matter of identity politics, 
but instead can literally have life or death stakes for users of color in our increas- 
ingly Al-driven future. 


Conclusion 


The hypervisibility/invisibility dialectic has historically sheltered the worst forms 
of racism in the United States and abroad. It has been endemic to visual media 
since their inception and is currently being cemented into the AI paradigms that 
we increasingly believe our going to usher in the next stage of human civiliza- 
tion. The approach that AI design takes to this dialectic thus ought to be a central 
battleground for anyone working towards the democratization of AI. We cannot 
have egalitarian or democratic technology if we hardcode pre-existing regimes of 
marginalization into our AI systems. While it is beyond the scope of this chapter 
to fully articulate an alternative to the hypervisibility/invisibility dialectic in AI 
systems, I think we look to some of the repeated refrains that have still yet to be 
initiated in Silicon Valley as a way to at least open the space where we might imag- 
ine a better alternative. In her WIRED article “How to Keep Your AI from Turn- 
ing into a Racist Monster,” Megan Carcia (2017) offers some common-sense ap- 
proaches that might be advocated for and instituted broadly to great effect. These 
systems ought to better empower users to analyze, debug, and flag problematic 
components of AI systems. This process not only crowdsources the labor, mak- 
ing it much more appealing to tech companies beholden to their shareholders, but 
also in the process educates users about the importance of this labor as well as the 
technological functioning of the systems they are helping to analyze. Second, Sil- 
icon Valley needs to hire more diverse computer programmers immediately and 
without making any more excuses. More diverse development teams are much 
more likely to recognize hardcoded marginalization prior to release and are sta- 
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tistically likely to generate greater profits. Third, empower more third-parties to 
engage in high-tech auditing of AI systems. Even if trade secrecy and fear of spam 
prevent the open sourcing of all code, certain trusted third parties more dedi- 
cated to advancing democracy and eradicating marginalization ought to be al- 
lowed access to these systems to help monitor hardcoded biases. Lastly, we ought 
to further develop a public discourse that demands egalitarian AI and institute 
an inter-company set of tools and standards to help better motivate companies 
and hold them accountable. While these are Carcia’s ideas, they are also ours, writ 
large, as they somehow echo across internet discourse ad nauseum without ever 
really being instituted in Silicon Valley. We might need to supplement this with 
more traditional grass-roots organizing and activism tactics to turn this polite 
request into a demand. Democracy often requires revolutionaries. 
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Mapping the Democratization of Al on GitHub 
A First Approach 


Marcus Burkhardt 


Over the course of the past 80 years the digital computer has radically changed the 
world we inhabit and the ways in which we relate to it and to each other. Likewise, 
computing has radically changed as well. At first, practical computing machines! 
carried individual names, but in the early 1950s proper names were quickly re- 
placed by series designators which are emblematic for the era of mainframe com- 
puters and time-sharing systems. The 1970s and 1980s gave rise to micro, home 
and personal computers as well as graphical user interfaces that became prevalent 
in the 1990s. With the rise of the World Wide Web (WWW) during this decade 
networked computing and networking changed the face of computing again. Re- 
gardless of the burst of the dot-com bubble the Web flourished throughout the 
early 2000s by being reframed as Web 2.0 and social web. During this period com- 
puting devices became increasingly mobile and desktop computers were super- 
seded by notebooks, smartphones and tablets, software gradually morphed into 
services and apps that rely on cloud infrastructures for distributed processing 
and storage. 

In June 2017 Sundar Pichai, CEO of Google Inc., declared yet another para- 
digm shift in the history of computing. Innovation should neither be driven by 
approaching problems as first and foremost digital nor mobile, but instead by tak- 
ing an AI first approach that is fueled by recent advances in the field of machine 
learning: “We believe smartphones should be smarter; they should learn from you 
and they should adapt to you. Technologies such as on-device machine learning 
can learn your usage patterns and automatically anticipate your next action sav- 
ing you time” (Pichai 2018). This statement reflects a central promise of machine 
learning applications, namely the ability to adapt to unforeseen futures without 
prior programming of a particular event: visual recognition of specific objects or 
persons that the program did neither “see” nor was trained on before, self-driving 
cars that can deal with new situations safely or chatbots that conduct conversa- 


1 Forthe concept of practical computing machines see Turing (1992). 
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tions with humans in an engaging manner. Conversely, the more such technol- 
ogies are built into the fabric of everyday life the more concerns are raised about 
their potential risks, e.g. biases and inequalities inherent in training data sets. 
As a result, machine learning models often produce (social) structures instead 
of adapting to them. Drawing on debates in critical algorithm studies this paper 
asks how machine learning and artificial intelligence as fields of technological de- 
velopment and innovation are in themselves structured. By providing an initial 
mapping of the coding cultures of machine learning and artificial intelligence on 
GitHub the paper argues for the importance to attend more closely to the hith- 
erto largely neglected infrastructural layers of code libraries and programming 
frameworks for the development of critical perspectives on the social and cultural 
implications of machine learning technologies to come. 


Democratization of Al 


In recent years the interest in artificial intelligence (AI) in general and in machine 
learning (ML) in particular skyrocketed once again. This ongoing development is 
to some extend driven by leading technology companies such as Google and its 
parent company Alphabet, Amazon Web Services (AWS), Facebook, IBM, Micro- 
soft etc. It rests upon the massive accumulation of data by these companies on 
the one hand and the establishment of large-scale cloud infrastructures as well 
as infrastructural services on the other hand. However, these companies do not 
simply contribute to the rapid technological developments in Al and ML, they also 
take part in shaping the imaginaries of smart, intelligent and autonomous tech- 
nologies as cornerstones of technological progress and enablers of social progress 
as well as economic prosperity for the years to come. 

Central to this is the recent push towards the democratization of artificial 
intelligence. Google (IANS 2017), IBM (Moore 2018), Apple (Simonite 2017) and 
Microsoft (n.d.) alike mobilize the notion of democratic AI to promote the shift 
towards ML driven technological innovation. In this context democratization 
can be understood “as the action/development of making something accessible 


m 


to everyone, to the ‘common masses” (Schmarzo 2017). For Microsoft this entails 
allowing “every person and every organization” (n.d.) to partake in the anticipat- 
ed benefits of AI whose effects will supposedly be as far-reaching as that of the 


printing press: 


With the advent of the printing press in the 1400s we have an explosion of infor- 
mation—the first democratizing event around access that made it possible for 
humans everywhere to start learning. Access to information has only spread from 
there. [...] The question is, how can we use all we have in terms of computational 
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power to solve this fundamental constraint? To make better sense of the world? 
That’s the essence of what Al is. It’s not about having Al that beats humans in 
games, it’s about helping everyone achieve more — humans and machines wor- 
king together to make the world a better place. (Ibid.) 


In view of the long history of exaggerated expectations of artificial intelligence, a 
certain skepticism regarding such a claim ofa revolutionary caesura is warranted. 
What is more important, nevertheless, is the question how Al is actually made 
accessible, i.e. how democratization is enacted. Once again, the case of Microsoft 
can serve as a paradigmatic example. The company pursues a four-fold strategy: 
(1) utilize AI to develop new modes of interaction with ambient computing tech- 
nologies; (2) build intelligence into every application; (3) allow developers to make 
use of these “intelligent capabilities”; (4) make computing infrastructure available 
as a service (ibid.). 

The promise of democratization is directed towards both technology devel- 
opers and their users. For developers this democratization entails the possibility 
to make use of AI in their own products and to partake in shaping the future of 
AI by having open or paid access to resources and services such as software li- 
braries, pre-trained machine learning models, frameworks, platforms and infra- 
structures. Users on the other hand are enlisted in the democratization of AI as 
beneficiaries of technologies that are “infused” (ibid.) with artificial intelligence 
and machine learning. In such technologies, intelligence is typically enacted as a 
wide range of limited scope capabilities and features: software applications that 
play chess or the game of go, cameras that recognize faces and take pictures when 
people are smiling, speakers that are capable to recognize, interpret and execute 
voice-based commands, cars that drive autonomously, chatbots that engage in en- 
tertaining, helpful or informative conversations with human beings etc. 


Practices of Machine Learning 


The capacities of ML technologies are designed as well as staged to astonish its us- 
ers. Among many researchers in the fields of science and technology studies and 
media studies Kate Crawford and Ryan Calo have argued for the “need to assess 
the impact of technologies on their social, cultural and political settings” (2016: 
311). It is, thus, important to gain a critical understanding of machine learning 
technologies in general and the current drive towards democratic Al in particular. 
Such a critical understanding is all the more significant since modern day deep 
neural networks as well as other ML approaches supposedly evade human com- 
prehension in principle. 
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Despite the secrecy imposed on algorithmic systems by corporations and the 
intrinsic opacity of machine learning systems (cp. Burrell 2016), there is much to 
know about ML technologies as Adrian Mackenzie eloquently argued: 


Machine learning is hardly obscure or arcane knowledge today. These techniques 
are heavily documented in textbooks [...], in how-to books [...], and numerous video 
and website tutorials, lectures and demonstrations [...]. We can more or less read 
about and indeed play about with implementations in software [...]. (2015: 431p.) 


Drawing on such diverse resources Mackenzie himself became a critical student, 
practitioner and investigator of multiple situated, hybrid machine learning prac- 
tices. In Machine Learners Mackenzie presents a hands-on inquiry of “machine 
learning as a form of knowledge production and a strategy of power” (ibid.: 9). 
Following Foucault’s notion of archaeology Mackenzie unfolds an archaeology of 
six operational formations that are central to ML: 


vectorization, optimization, probabilization, pattern recognition, regularization, 
and propagation. These generic operations intersect in a diagram of machine le- 
arning spanning hardware and software architectures, organizations of data and 
datasets, practices of designing and testing models, intersections between scien- 
tific and engineering disciplines, and professional and popular pedagogies. (Ibid.: 
18) 


The approach to inquire machine learning not from afar, but by “learning to ma- 
chine learn” (ibid.: 18) resonates well with Wendy Chuns claim “that software can 
only be understood in media res” (Chun 2008: 323). In the middle of things that 
constitute ML today Mackenzie engages not only with textbooks, tutorials, math- 
ematical formulae, algorithms, and data sets, but also with numerous software 
libraries. While code libraries and programming frameworks are often refer- 
enced as crucial infrastructural elements of today’s software culture they are 
hardly studied closely in critical research (cp. Berry 2011; Marino 2014). Machine 
Learners, too, acknowledges the importance of code libraries frequently, but does 
not research them in detail. In passing, however, Mackenzie offers some valu- 
able insights into how code libraries and programming frameworks shape ma- 
chine learning practices by crystallizing “a repertoire of standard operations, 
patterns, and functions for reshaping data and constructing models that classify 
and predict events and associations among things, people, processes, and so on” 
(Mackenzie 2017: 23). Their architecture “classifies and orders” (ibid.: 77) machine 
learning asa field of interrelated practices as much as a domain of knowledge pro- 
duction. They constitute the “accumulating sediment of coding and related data 
practices [...] in which machine learners take root” (ibid.: 23). For Mackenzie the 
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implementation and widespread use of code libraries are, thus, articulations of 
contemporary coding cultures. 

Code libraries provide resources that developers can draw from and build 
upon. By offering predefined functions and functionalities they relieve develop- 
ers from building software from the ground up. At the same time code libraries 
impose their functional logics and practical affordances on developers. In this re- 
gard code libraries can be considered as media of “co-operative action” consisting 
of accumulative resources that can be laminated into operational software (Good- 
win 2018: 129). However, code libraries are sites of cooperation as well. They are 
created, contributed to, maintained, updated and deprecated in the “recursive 
publics” (Kelty 2008: 28) of code-sharing platforms such as GitHub.” 

As the “Facebook of coding” (Wulf 2017) GitHub today hosts more than 96 mil- 
lion repositories (GitHub 2019), most of them contain codes or coding related re- 
sources. For today’s coding culture GitHub serves as a center of gravity for hosting 
open and closed source software projects as well as for finding, contributing and 
debating code resources including libraries and frameworks (Mackenzie 2018). 
This applies in particular to current developments in ML. Yet, how exactly does ML 
as a practice and field take shape on GitHub? How do algorithmic techniques for 
ML, data sets, machine learned models and other resources circulate on GitHub? 
Which actors are involved in shaping this space of cooperation, exchange and ne- 
gotiation and which strategies of power are deployed? Or to put it differently: How 
is the democratization of AI enacted on the infrastructural level of code and cod- 
ing sharing practices? 


Mapping the Democratization of Al in Code 


For mapping the numerous heterogenous articulations and manifestations of 
machine learning and artificial intelligence on GitHub researchers can make use 
of the application programming interface (API) provided by the platform which 
allows for the retrieval of repository metadata that match certain search criteria, 
like specific keywords contained in the repositories description or topics assigned 
to the repository by its creator.” However, the GitHub search API poses certain 
restrictions: it allows only for a limited number of keywords per query and returns 
only a maximum of 1.000 results for each query. While this might be sufficient 
to explore the most visible projects on GitHub for a specific keyword in regard to 


2 Foradiscussion of the relevance of version control systems and GitHub for contemporary coding 
cultures see Burkhardt (2019) and Mackenzie (2017, 2018). 

3 A similar mapping of cultures of coding and sharing has been undertaken by Kollanyi (2016) in 
respect to Bots. 
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the number of forks or stars a repository received, this limitation disregards the 
variety of keywords associated with ML and Al as well as the long tail of software 
development and code sharing practices in the field of ML. In order to map the 
enactments and materializations of machine learning more comprehensively a 
search tactic must be deployed that charts the space of ML and AI related repos- 
itories iteratively by identifying and working through an extensive list of search 
terms and by restricting the search parameters (programming languages, num- 
bers of forks, received stars and size) step by step: 


> search terms 

— search terms + programming language 

— search terms + programming language + fork count 

> search terms + programming language + fork count + star count 

> search terms + programming language + fork count + star count + size 


This search tactic provides a snapshot view of a dynamically changing environ- 
ment. The results remain incomplete, but are more comprehensive than the de- 
fault limit of 1.000 results per query.* What is left out and remains invisible in 
principle are all the private repositories hosted on GitHub. 

Assembling a dataset is all but the first step in mapping the enactments of 
machine learning in contemporary coding culture. The dataset contains 211.802 
unique repositories.* Among those 41.818 contain artificial intelligence‘ as a key- 
word and 103.344 machine learning’. Remarkably only 3.064 repositories refer- 
ence both conceptual fields although in public discourse the current summer of Al 
is largely attributed to developments in the area of machine learning. 


4 Certain gaps in the result sets can be pinned down precisely: The result limit is exceeded for 
programming languages like Python that have neither been forked nor received a star and are 
relatively small (in the case of Python it is 0, 1, 2, 3 ..., 11 KB). However, in other cases gaps emerge 
as inconsistencies between the supposed number of results returned by the GitHub API and the 
actual number of results retrieved. 

5 Search terms: ai, dl, nn, ml, artificial-intelligence, artificialintelligence, artificial intelligence, 
machine learning, machinelearning, machine-learning, machine intelligence, deep learning, 
deep-learning, deeplearning, neural nets, neuralnets, neural-nets, neural net, neuralnet, neu- 
ral-net, neural networks, neuralnetworks, neural-networks, neural network, neuralnetwork, 
neural-network, neural, bigdl, caffe, caffe2, cntk, coreml, deeplearning4j, keras, lasagne, mlib, 
mlpack, moa, mocha.jl, mxnet, neon, Paddle Paddle, pylearn, pylearn2, pytorch, scikit-learn, sho- 
gun, singa, tensorflow, tflearn, theano. 

6 Variations considered: artificial intelligence, artificial-intelligence, artificialintelligence, ai. 


7 Variations considered: machine learning, machine-learning, machinelearning, ml. 


Mapping the Democratization of Al on GitHub 


215 


Language % 

Python 47.74 

HTML 1.25 

5 | Java 8.46 

g 1“ | Matlab 6.90 
i 1“ | JavaScript 5.33 
z IR 3.19 
: x| cH 3.09 
So A 9 ER = € er Jupyter Note- 2.18 

oe s $ we oi Poa sY book 

giie bi c 159 

MATLAB 1.10 
Total 90,83 


Fig. 1: Distribution of top 10 programming languages used in the retrieve repositories. 


Machine learning and artificial intelligence materializes in a range of program- 
ming languages. Python (together with Python-based Juypter Notebooks) is by far 
the most often used programming language for ML. Remarkably this is followed 
by eleven percent HTML repositories. As will be discussed later this is because not 
all repositories on GitHub contain code as a resource. Many contain informational 
and educational resources such as tutorials, book manuscripts, research papers, 
course materials or collections of links. 

The popularity or relevance of repositories can be inferred from the number of 
forks as well as stars they received. In the context of GitHub in particular and the 
Git version control system in general forks are copies of a repository. Forking con- 
stitutes a central practice in Git-based collaboration: “Most commonly, forks are 
used to either propose changes to someone else’s project or to use someone else’s 
project as a starting point for your own idea” (GitHub a). Stars on the other hand 
are platform specific indicators of some kind of interest of users in a repository: 

“You can star repositories and topics to keep track of projects you find interesting 
and discover related content in your news feed” (GitHub b). 
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Fig. 2: Distribution offorks by repository 


The distribution of stars and forks across repositories adheres to the power law. 
Only few repositories gain high visibility, while the largest number of reposito- 
ries has no forks and received little to no stars. The long tail of machine learning 
is a space of testing out, experimenting with and learning by doing. This space, 
however, is also populated with course assignments, original research in progress, 
personal collections of resources on machine learning, and new, emerging, failed 
or abandoned code libraries. Among the more than 200.000 unique repositories 
that were retrieved for this article less than 0.8 percent have more than 100 or 
more forks and only 1.8 percent received 100 or more stars. 


Total number 2100 forks 2100 stars 
Artificial intelligence repositories 41.818 284 593 
Machine learning repositories 103.344 799 1.677 
Deep learning repositories 33.749 857 1.842 
Neural network repositories 46.681 558 1.365 
All repositories 211.802 1.751 3.951 


High level observations on the use of programming languages or the distribution 
of popularity offer some initial insights into how artificial intelligence and ma- 
chine learning is articulated on GitHub. A more detailed analysis ofthe 200 most 
often forked and starred repositories, however, reveals the heterogeneity and 
diversity of resources developed, published, collaborated, maintained, debated, 
updated and downloaded. Among those top 200 repositories are 42 that can be 
categorized as code libraries or programming frameworks. However, more than 
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fifty percent contain informational and educational resources. About 15 percent 
of the repositories provide reference implementations for algorithms, machine 
learned models for specific application domains or software applications based on 
machine learning. And only few provide infrastructural resources, programming 
languages, experiments or datasets for ML. 


Type # 
Informational/Educational 15 
resource 

Library/Framework 42 
Algorithms/Models/Applications 31 
Infrastructure/Optimization 8 
Experiments 2 
Languages 1 
Datasets 1 


The informational and educational resources address a range of distinct audienc- 
es like machine learning beginners, learners of a framework, machine learning 
professionals as well as researchers. As a result, some repositories provide re- 
sources on the fundamentals of machine learning, materials for online courses, 
tutorials for different code libraries or introductory book publications. Others are 
framed as collections, comprehensive lists or curated lists on machine learning in gen- 
eral, state of the art research or more specific topics like infrastructures. In part 
these resources are created, assembled and provided by individual developers, re- 
searchers or authors. Yet, many informational and educational repositories are 
created and maintained by corporations to educate users about and recruite them 
for their products. The repository amazon-sagemaker-examples* released by Amazon 
Web Services Labs for example contains basic information and training materials 
on using the companies SageMaker platform for the training, optimization and 
deployment of machine learning models. Here the infrastructural complexities 
of machine learning practices become visible. Understood as a mode of databased 
programming machine learning relies on large scale processing capacities that 
today are provided as cloud computing services—a market dominated by global 
technology corporations like Amazon, Microsoft, Google, IBM etc. (Flexera 2019). 
What is more, engaging with the rich diversity of educational and informational 
resources reveals the central role of programming libraries for coding cultures. 
Learning to machine learn is deeply intertwined with learning to make use of code 


8 https://aws.amazon.com/de/sagemaker/ 
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libraries and frameworks. While there are repositories dedicated to do “Machine 
Learning From Scratch”, educational resources promising information on how to 
do machine learning with library x or framework y are more common. 

The variety of repositories with informational resources is reflected in the het- 
erogeneity of code repositories. Today, by far the most popular repository in ma- 
chine learning and artificial intelligence is the TensorFlow framework.” Initially 
developed by Google Brain for internal use the framework has been released in 
2015 as open source by Google and is since maintained by the company, but it has 
attracted a large number of external contributors as well. TensorFlow is a frame- 
work for deep learning that mainly supports the training and deployment of neu- 
ral networks and is, thus, dedicated to a specific paradigm of machine learning 
that has become dominant throughout the past decade. Other popular libraries 
such as scikit-learn" focus on other areas of machine learning and in consequence 
support a broader range of approaches.” While it is possible to implement basic 
neural networks with scikit-learn they play a somewhat marginal role in this li- 
brary. This is underlined by the missing support for deep and reinforcement learn- 
ing as well as the use of GPU hardware (scikit-learn 2019, 5 and 7). TensorFlow and 
scikit-learn, thus, are different articulations and materializations of machine 
learning. Yet, they are similar in that they can both be considered as general-pur- 
pose frameworks, i.e. they are not explicitly focused on specific application do- 
mains. 

A number of special-purpose machine learning libraries have gained a relative- 
ly high popularity as well. Among them are the Unity ML-Agents Toolkit”, OpenCV", 
and the ChatterBot” and Rasa’ frameworks. The ML-Agents Toolkit allows the im- 
plementation of so-called machine learning agents in the Unity game engine. As 
a plugin for a development environment for games the toolkit is of course aimed 
at this application area, but also allows for the development of algorithms for ro- 
botics or the training of self-driving cars in virtual environments (cp. Unity n.d.). 
Chatterbot and Rasa are frameworks for the implementation of AI-based chatbots. 
Machine learning here articulates itself as the use of pretrained models for nat- 
ural language understanding and dialogue management as well as the ongoing 


9 https://aws.amazon.com/eriklindernoren/ML-From-Scratch 

10 https://github.com/tensorflow/tensorflow 

11 https://github.com/scikit-learn/scikit-learn 

12 Fora comprehensive overview of machine learning paradigms and approaches see Domingos 
(2015). 

13 https://github.com/Unity-Technologies/ml-agents 

14  https://github.com/opencv/opencv 

15 https://github.com/gunthercox/ChatterBot 


16 https://github.com/RasaHQ/rasa 
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improvement of such models based on training data gathered in beta tests and 
during real world deployment. By providing code resources for the development 
of domain specific software, such frameworks are not for machine learning in 
general, but have some element of machine learning build in. This raises the ques- 
tion how special-purpose frameworks structure machine learning practices in 
specific ways as well as how they prescribe the operational logics and performanc- 
es of applications created with them. In the case of chatbots that is to ask how 
conversationality and communicability is inscribed by such frameworks. 

In addition to general-purpose and special-purpose libraries at least a third 
type can be distinguished which could be called meta-libraries. A popular example 
for this kind of library is Keras”. It is built on top of the deep learning frameworks 
TensorFlow, Theano" and CNTK” and provides a unified interface to a multiplicity 
of different code libraries. Keras, thus adds an abstraction layer which serves as 
frontend to the backend of multiple deep learning libraries. Two of those libraries 
are developed by global corporations—Google in the case of TensorFlow and Mi- 
crosoft in the case of CNTK (Cognitive Toolkit). The third framework was mainly 
developed in academic contexts and is maintained by the Montreal Institute for 
Learning Algorithms. The end of its active development has been announced in 
2017 citing among other reasons that “strong industrial players are backing dif- 
ferent software stacks in a stimulating competition” (Bengio 2017). Indeed, large 
global digital technology companies compete in the space of open source machine 
learning libraries with respective frameworks. PyTorch?° (Facebook), Neo-AI-DLR™ 
(Amazon), Aerosolve”” (AirBnB) as well as the already mentioned frameworks Ten- 
sorFlow (Google) and CNTK (Microsoft) are just a few examples. The release of 
machine learning libraries in open source by Google, Facebook, Amazon, Micro- 
soft etc. can be understood as an effort towards the democratization of AI. At 
the same time, the open source coding culture has become a space of corporate 
intervention and competition. How this affects the future of machine learning 
technologies as well as the ways in which machine learning will be built into the 
fabric of our technological world are questions that remain unanswered. Critical 
research, thus, needs to attend even more closely to the logics and politics of code 
libraries and programming frameworks. 


17  https://github.com/keras-team/keras 
18 https://github.com/Theano/Theano 
19 https://github.com/microsoft/CNTK 
20 https://github.com/pytorch/pytorch 
21 https://github.com/neo-ai/neo-ai-dlr 
22 https://github.com/airbnb/aerosolve 
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On the Media-political Dimension 
of Artificial Intelligence 
Deep Learning as a Black Box and OpenAl' 


Andreas Sudmann 


Neither machines nor programs are black boxes; they are artifacts that have been 
designed, both hardware and software, and we can open them up and look inside. 
(Allen Newell/ Herbert A. Simon 1997 [1976], 82) 


What does it mean to critically explore the media-political dimension of modern 
Artificial Intelligence (AI) technology? Rather than examining the political as- 
pects of specific Al-driven applications like image or speech recognition systems, 
the main focus of this essay is on the political implications of APs technological 
infrastructure itself, especially with regard to the machine learning approach that 
since around 2006 has been called Deep Learning (in short: DL, also known as 
the simulation of neural networks or Artificial Neural Networks—ANNS). First, 
this essay discusses whether ANN/DL has to be perceived as a fundamentally 
opaque or ‘black box’ technology, perhaps inaccessible or only partially accessible 
to human understanding. Second, and in relation to the first question, the aim 
is to take a critical look at the agenda and activities of a research company called 
OpenAl that purpotedly promotes the democratization of AI and tries to make 
technologies like DL more accessible and transparent. Obviously, such idealistic 
claims should not simply be taken for granted, especially if one takes into account 
the large amount of capital invested in a company like OpenAl. For example, strat- 
egies like open-sourcing AI seem more likely to serve the purpose of demonstrat- 
ing those companies’ technological potential, to one-up each other, and/or to at- 
tract rare talents. But perhaps even more important than simply questioning the 
authenticity or ideological implications of such claims, we have to address more 
fundamental problems here: How can one contribute to the transparency and ac- 


1 This is a slightly revised and extended version of an essay that has been published under the 
same title in: Ramón Reichert, Mathias Fuchs (Ed.), Rethinking Al. Neural Networks, Biopolitics and 
the New Artificial Intelligence, Digital Culture & Society 4/1, 2018, pp.181-200. 
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cessibility of a black box that—perhaps—cannot be opened at all? And can there be 
a democratization of AI without a democratization of data in general? 


1. The so-called “Al revolution” 


Before addressing these questions, it is important to recapitulate how DL recently 
managed to become the dominant paradigm of AI. An event of major importance 
in this respect happened in 2012. Back then, three scholars from the University of 
Toronto, Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton, for the first time 
effectively trained a so-called Convolutional Neural Network, which is a special 
variant of a traditional Artificial Neural Network (ANN) optimized for image and 
object recognition, on the basis of the now famous database ImageNet as well as 
on fast parallel-organized GPU processors (cf. Huang 2016). The substantial em- 
ployment of GPUs made all the difference: The Toronto team was able to reduce 
the error rate of previous approaches in image recognition by more than the half.” 
While this increase may not sound very impressive, it was big enough to attract 
the attention of leading IT companies like Google and Facebook, which quickly 
hired leading scientists like Yann LeCun and Geoffrey Hinton and also acquired 
AI start-up companies such as DNNResearch and DeepMind. The strong interest 
of these companies in DL technology was no surprise since they were already har- 
nessing big data, and now would have access to a powerful technology to process 
and harness it intelligently (cf. Reichert 2014). For example, thanks to DL it is pos- 
sible to automatically tag images uploaded by users on social media-platforms, or 
to analyse consumer behaviour to generate individualized ads, or make person- 
alized recommendations. Of course, there are many other application areas for 
which DL/ANN technology is currently used: e.g. to process the sensory data of 
self-driving cars, to analyse data for stock market predictions, or for optimised 
machine translations, etc. In general, DL algorithms are a universal instrument 
for pattern-recognition and prediction tasks, an effective tool to manage uncer- 
tainty and fuzziness in data content (Goodfellow/Bengio/Courville 2016). 
However, it took a while until modern machine learning algorithms were able 
to unfold their potential. Some of the technological essentials of DL were already 
developed in the 1940s and 1950s (cf. Sudmann 2016). Already back then, the basic 
idea of this AI paradigm was to develop a computer system that should be able 
to learn by observation and experience to solve specific problems or fulfil certain 
learning tasks, without having concrete rules or theories guiding the process 


2 Over a period of just seven years, the accuracy in classifying objects in the dataset rose from 
71.8% to 97.3%. Not least due to his high value, 2017 was the last year of this famous competition. 
Cf. Gershgorn (2017) for the details. 
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(Mitchell 1997). This is the basic approach of every existing machine learning sys- 
tem, as opposed to so-called symbolic, rule-based forms of AI systems whose in- 
telligent behaviour typically is more or less hand-coded in advance (Boden 2014). 
Even though there are many ML approaches out there, it recently turned out that 
DL methods are the most effective ones, at least with regard to key areas of AI 
research like natural language processing and computer vision. 

Very broadly speaking, one of the key characteristics of DL is that it is a class 
of techniques that are loosely inspired by the structure and learning processes of 
biological neural networks (Alpaydin 2016). As with other machine learning tasks, 
DL algorithms learn by analysing thousands, if not millions of training data on 
the basis of thousands or even millions of iterations up until the very moment the 
system is able to predict unseen data correctly. Yet what distinguishes DL from 
other machine learning approaches is the hierarchical distribution of its learning 
process. DL technology simulates networks typically consisting of millions of ar- 
tificial neurons that are organized on different layers—an input, an output and a 
flexible number of intermediate hidden layers (Trask et al. 2015). If a network is 
called deep, it has at least two or more intermediate layers that process the infor- 
mation through the network. On the lowest level of layers, the network analyses 
very simple forms of input (for example, lines and edges, in case of visual data) 
and forwards this information to the next level of layers, which processes more 
complicated forms (for example, parts of the object like a face or legs) and again 
forwards this information to the next highest level, all the way up to the final lay- 
er, the output layer, which than can predict if a certain unknown input correctly 
matches with a certain output (does the image show a certain object or not?). 


2. The Media-politics of Deep Learning 


It is also not very surprising that the current AI boom quickly started to attract the 
attention of the humanities and social sciences, whereas before 2016 many disci- 
plines outside the hard sciences more or less ignored machine learning technolo- 
gies like DL/ANN. Of course, there has been a long tradition of an inter- and trans- 
disciplinary debate concerning the potentials and limits of AI (cf. Weizenbaum 
1976, Searle 1980), yet those discussions typically did not address the technology of 
ANN in any great detail. There are some important exceptions to highlight in this 
respect, especially the discourses of philosophy of mind as well as cognitive psy- 
chology, which very early developed an interest in both the symbolic and connec- 
tionist forms of AI (cf. Horgan/Tienson 1996, Haugeland 1997). Furthermore, even 
in the field of media studies, one can find a few cases where scholars have been 
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discussing ANN technology.’ One example is the introduction of Computer als Me- 
dium (1994), an anthology co-edited by Friedrich Kittler, Georg Christoph Tholen, 
and Norbert Bolz. Interestingly enough, the text (written by Bolz only) somehow 
captures the epistemic relevance of the connectionist paradigm of AI, yet does so 
without exploring the details of its media-theoretical or -historical implications. 

In the last two years or so (more or less after the success of DeepMind’s Alpha- 
Go), the situation has changed significantly. More and more books and articles are 
published, for instance in the humanities that tackle the topic of AI in general and 
DLtechnology in particular (Sudmann 2016, Pasquinelli 2017, Finn 2017, McKenzie 
2017, Engemann/Sudmann 2018). For example, Pasquinelli (2017) recently wrote a 
short essay on ANN from a historical and philosophical perspective, arguing (with 
reference to Eco and Peirce) that the technology can only manage inductive rea- 
soning, whereas it is incapable of what Peirce calls abductive reasoning. Further- 
more, there are authors like Nick Bostrom (2014), Ed Finn (2017), or Luciano Floridi 
(2017) who are already very much engaged in the political and ethical discussion 
of current AI technologies. For example, Nick Bostrom’s book Superintelligence: 
Paths, Dangers, Strategies (2014) attracted much public attention, partly because of 
its alarmist thesis that the technological development of a super machine intelli- 
gence is mankind’s greatest threat, which was later echoed by a twitter post from 
Elon Musk. Yet, not everyone concerned with the political and ethical aspects of 
AI shares these apocalyptical views. Luciano Floridi, for instance, is convinced 
that humankind is able to handle an Al-driven society as long as we instantiate a 
“system of design, control, transparency and accountability overseen by humans” 
(2017: online). 

Yet, what is still widely missing in the intellectual debate is a discussion of 
AI/DL from a decidedly media-political perspective. But what does such a focus 
involve, and why do we need it in the first place? To begin with, there are—of 
course—many different ways to think about the media-political dimension of AI 
in general and DL in particular. For example, one possible approach would be to 
claim that “media politics” as an analytical agenda is concerned with the media- 
tion of politics and/or the historical relationship of media and politics (cf. Dahl- 
berg/Phelan 2011). Based on such an account, one could ask, for instance, how AI/ 
DL technology inscribes itself in relations of media and politics, or how it partic- 
ipates in the mediation of politics. In both cases, we might assume that a) media/ 
mediation and politics are basically distinct concepts, and that b) possible analyt- 
ical perspectives are very much shaped and guided by our basic understanding 
of these terms in the first place (including to perceive AI technology itself as a 
medium). 


3 Ofcourse, there are some more publications from a media studies perspective that deal with Al 
technology in general, for example: Dotzler 2006. 


On the Media-political Dimension of Artificial Intelligence 


Yet another possible approach would be to claim that media have an inherently 
political dimension (and, similarly, one could claim that nothing political exists 
outside a medium or certain media). Still, the question remains if this is true for 
every concept of media or medium or just particular ones. But this is a rather the- 
oretical discussion, since most concepts of media politics are more or less based 
on a traditional understanding of media as mass or popular media (cf. Zaller 1999, 
Dahlberg/Phelan 2011).* 

In the context of this essay, my understanding of a media-political account, 
however, is a broader and in a certain light more basic one. Such a theoretical per- 
spective, as I like to conceptualize it, is not so much concerned with the represen- 
tations or visible interfaces of AI, but more interested in the political implications 
and effects of the medial infrastructure and entities that generate and shape the 
technology (also regardless of particular ‘use cases’).° In other words, what I am 
interested in are—what I like to call—the infra-medial conditions of modern AI tech- 
nology and their political dimension.‘ For me, every entity involved in the process 
of generating and shaping AI technology can generally be perceived as a medi- 
ator (cf. Latour 2005). And generally, every mediator of technology also matters 
in political terms. However, not every mediator can be equally conceptualized as 
a medium, at least not if one applies a more narrow understanding of the term, 
for example, to regard media as entities or dispositifs that enable communication 
or that store, process, or transmit information.’ For this very reason, it generally 
makes sense to differentiate between the concepts of mediator(s) and medium/ 
media. 

Yet while I argue that we need such a distinction, Iam nevertheless quite scep- 
tical about using a stabile concept of the term “medium” or “media” (even though 
it would make the task of differentiating both terms much easier). In my mind, in 


4 Accordingly, one possible approach would be to examine the politics of representation of differ- 
ent Al technologies in popular media like film and television. 


un 


For a similar account, using the term “media infrastructures” as a critical concept, cf. Parks/ Sta- 
rosielski (2015). 


lon 


Such a perspective is not directly concerned with a specific theoretical framework. Generally, this 

focus is compatible with many analytical approaches like media archaeology, historical episte- 
mology, or actor-network theory. The prefix “infra” highlights to look at media (operating) ona 

level that is generally invisible (for example on the level of code). 

7 This is a different account of how media can be conceptualized with reference to Latour’s dif- 
ferentiation between “mediators” and “intermediaries”. For Latour, an “intermediary [..] is what 
transports meaning or force without transformations” opposed to “mediators” that “transform, 
translate, distort, and modify the meaning or the elements they are opposed to carry” (2005: 39). 
Intermediaries function in a certain sense as black boxes, since their input allows you to predict 

the respective output (without having knowledge of the object’s internal operations). In opposi- 
tion to that, in case of mediators, despite the specific nature of a certain input, it is never easy to 

predict the respective output (ibid., cf. also Thielmann 2013). 
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order to make sense of our empirical world’s entities (including the immaterial 
world of our thoughts), the terms media and/or medium are more productive in 
analytical terms if one regards them as flexible epistemological-heuristic rather 
than fixed categories. Accordingly, media theory, as I advocate it, can be under- 
stood as the general task to explore in what different ways the world is perceivable 
as a medium or as media (with certain characteristics, functions, inscriptions) 
rather than simply acknowledging that everything out there in the world depends 
on media or a medium in some ontologically stabile sense (as a precondition of 
entities to be visible, to be perceivable, or to have a certain form etc.). Hence, even 
though I opt for a concept of media politics that examines the constitutive role of 
mediators (and of specific media), I still advocate a rather open analytical focus that 
leaves room for very different perspectives. 

The latter position seems also to be an instructive approach with regard to the 
political dimension of media politics. For example, we can quite easily claim that 
almost everything about Al is political, especially if one believes that AI/DL tech- 
nology affects every aspect of our social and cultural existence. At the same time, 
the political challenges that AI and DL technology hold for us are very different 
in nature (the existential threat of Al-driven autonomous weapon systems, Al’s 
influence on the future of work, etc.), which is why we cannot simply refer to a 
master account of political theory suitable for each and every problem. To provide 
another example: When designing autonomous weapon systems, there is obvi- 
ously a strong political interest involved in keeping hidden the mediators of the 
production of these technologies, as well as their functioning. On the other hand, 
with regard to the question of how Al affects the future of work, for example, it 
may be more important to make all mediators involved in this process as transpar- 
ent and accessible as possible. 

Such a plea does not mean that “anything goes” in terms of how we should 
address the politics of AI/DL. Instead, I argue that one should—first of all—try to 
explore how contemporary AI technologies emerge as political phenomena (before 
we apply a certain political theory to Al). This focus entails many relevant aspects, 
including the analysis of ways in which computer scientists themselves conceptu- 
alize AI technology as a political subject. 

In this context, one should also keep mind that the subjects of machine learn- 
ing in general and ANN/DL in particular are, again, still an unknown territory 
for most scholars working in the humanities or social sciences, even if they have 
already studied AI. What this basically means is that it might take a while until 
disciplines like media or cultural studies are really able to evaluate the relevant 


8 There are perhaps many approaches to justify such a concept of media-thinking. Obviously, we 
can again refer to Latour’s category of “mediators”. A similar theoretical reference in this context 
is also Vogl’s term of “becoming-media” (2008). 
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political and/or ethical aspects of DL’s technologies and infrastructures. Obvious- 
ly, this problem is also a central factor in discussing AI/DL as a black box and in 
evaluating projects like OpenAl at some point. For many scholars in the field, it 
is one of media studies’ central tasks to focus on processes of knowledge transla- 
tion or transformation and to analyse, from a kind of meta-perspective, how the 
knowledge of one discipline is used, adapted, and reconfigured by another disci- 
pline (cf. for example Bergermann/Hanke 2017: 127). But how can media studies 
provide relevant insights into the black box problem of AI/DL if even computer 
or data scientists have profound trouble dealing with it? Obviously, media stud- 
ies has nothing or little to contribute towards opening this black box in technical 
terms, yet it can perhaps shed light on different aspects: for example, exploring 
the problem’s complex network of socio-cultural conditions, implications, and 
effects. Furthermore, media studies can—of course—critically investigate how 
data scientists treat AI and its black box problem as a political concern. But in 
order to do so, let’s recapitulate what it means—or better—what it could mean to 
perceive certain entities of our empirical worlds as black boxes. 


3. Deep Learning: A Black Box that Cannot be Opened? 


There are some debates going on about the exact origins of the term black box. 
Philipp von Hilgers has explored the history of the concept and traced it back to 
the history of World War II, more precisely to the technology of the magnetron 
(Hilgers 2009). Since then, the concept has been applied and specified in very 
different contexts with opposed meanings. On the one hand, it can refer to the 
data-monitoring systems in planes or cars; on the other hand, it encompasses sys- 
tems whose inner operations are opaque or inaccessible and thus only observable 
by their inputs and outputs (cf. Pasquale 2015: 3). One early definition of the term 
black box has been provided by Norbert Wiener, in a footnote of the preface added 
to the 1961 edition of his famous book Cybernetics (1948): “I shall understand by a 
black box a piece of apparatus [...] which performs a definite operation [...] but 
for which we do no necessarily have any information of the structure by which 
this operation is performed” (p. xi).’ Last but not least, as Latour explains, one 
has to consider that the operations of science and technology always have a black 
boxing effect: “When a machine runs efficiently, when a matter of fact is settled, 
one need focus only on its inputs and outputs and not on its internal complexity. 
Thus, paradoxically, the more science and technology succeed, the more opaque 
and obscure they become” (Latour 1999: 99). Prima facie, this also seems to be true 


9 Towards this definition, cf. W. Ross Ashby’s 1956 book An Introduction to Cybernetics. 
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for DL systems. And yet, as opposed to other forms of technology, the case of DL 
technology seems to be different. 

Typically, independent of the black boxing effect just mentioned, many, if not 
most operations of technology used in practice are in one way or the other acces- 
sible to human understanding and explanation. In contrast, DL algorithms seem 
to be a black box that cannot be opened. At least this is what several experts cur- 
rently identify as one of Al’s biggest problems. But is it actually true that DL is a 
fundamental opaque technology and if so, to what degree? And even if this is the 
case, can’t we simply accept ANN to be an opaque technology as long as it works 
smoothly? The latter question may appear less complicated to answer than the first 
one. In fact, there already seems to be a large consensus among many scientists, 
politicians, and leading IT companies to develop a responsible or ethical AI, and 
making the technology more accountable is one essential part of this endeavor. 

This broad consensus is, of course, no surprise. It’s one thing if an AI/DL sys- 
tem comes up with a disappointing movie recommendation, but if we use intelli- 
gent machines for more serious matters concerning questions of life or death, the 
story is a completely different one. As Tommi Jaakkola, computer scientist at MIT, 
recently pointed out: “Whether it’s an investment decision, a medical decision, 
or maybe a military decision, you don’t want to just rely on a ‘black box’ method” 
(Knight 2017). 

For this reason, it might not be enough knowing that the predictions of your 
Al/ DL system are sufficiently accurate. Furthermore, you want to understand 
why the system comes up with a certain prediction. Both aspects seem highly rel- 
evant to secure trust in an Al-driven decision. Yet, to grasp the meaning of Al’s 
prediction models seems to be rather challenging. To illustrate this last point: Re- 
cently, researchers at the Icahn School of Medicine at Mount Sinai developed an 
AI program called “Deep Patient”. The system is able to identify different forms 
of diseases and even early indications of psychiatric disorders like schizophrenia 
astonishingly well, yet they still do not have a clue how this is possible. Of course, 
Deep Patient can be of great help for doctors, but they need the system to provide 
a rationale for its predictions, so that they have a solid reference for the medical 
treatment of their patients. “We can build these models,” as Joel Dudley, the direc- 
tor of biomedical informatics at the Icahn School of Medicine, explains, “but we 
don’t know how they work” (Knight 2017). 

In the following, I discuss in how far this assumption, which regularly appears 
in current AI discourses, is somehow misleading and in need of clarification. First, 
one should keep in mind that the math behind current DL technology is rather 
straight forward (cf. Goodfellow/Bengio/Courville 2016). Ultimately, it is a mat- 
ter of statistics, albeit an advanced form of statistics. This aspect is important to 
highlight since we can observe a general tendency towards mystifying DL that is 
counterproductive and needs to be contained. Secondly, many experts stress that 
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ANN are in fact an accessible technology, especially if one compares them with bi- 
ological neural networks. For example, Roland Memisevic, chief scientist of the 
Toronto-Berlin-based DL company TwentyBN, points out that “DL algorithms are 
at least way more accessible than the human brain, where the neuronal activity 
patterns as well as the transformations effected by learning are, even today, still 
very much opaque. In contrast, if one looks at an ANN model, you can record, ob- 
serve, measure everything, down to the smallest detail. For example, it is easy to 
find out which features have falsely resulted in a dog being labelled as a cat, be- 
cause certain ear shapes might again and again lead to certain misclassifications” 
(Memisevic 2018, my own translation). However, what indeed is difficult to under- 
stand is the interplay of the artificial neurons, as Memisevic agrees, “since having 
such a great number of neurons that are active in parallel, one is confronted with 
emergent phenomena, whereby the whole encompasses more than the sum of its 
parts” (ibid.). 

Thus, while it is certainly true that computer scientists have to deal with what 
is commonly labelled the interpretability problem of DL, it is not as fundamental 
as it is often described in the current discourse (cf. Knight 2017). And, not sur- 
prisingly, computer scientists inside and outside the tech industry are currently 
very busy trying to come to terms with this interpretability problem. In fact, re- 
searchers have already developed a number of approaches to better understand 
and reverse-engineer DLs prediction models. 


4. Strategies of Explainable Al (XAI) 


One example to make not only ANN but machine learning technologies in general 
more accessible is the program Local Interpretable Model-Agnostic Explanations 
(LIME), developed by Marco Tulio Riberio, Sameer Singh, and Carlos Guestrin. 
The authors describe it as “a technique to explain the predictions of any machine 
learning classifier, and evaluate its usefulness in various tasks related to trust” 
(Riberio/Singh/Guestrin 2016). The basic idea behind LIME is to change different 
forms of inputs (e.g. texts or images) to the AI system in such a way that one can 
observe if and how these variations of the input have an impact on the output. A 
recent article in the journal Science explains how LIME works in practice, with ref- 


erence to an ANN that is fed with movie reviews: 


[A neural network] ingests the words of movie reviews and flags those that are po- 
sitive. Ribeiro’s program, called Local Interpretable Model-Agnostic Explanations 
(LIME), would take a review flagged as positive and create subtle variations by de- 
leting or replacing words. Those variants would then be run through the black box 
to see whether it still considered them to be positive. On the basis of thousands of 
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tests, LIME can identify the words—or parts of an image or molecular structure, or 
any other kind of data—most important in the Al’s original judgment. The tests 
might reveal that the word ‘horrible’ was vital to a panning or that ‘Daniel Day Le- 
wis’ led to a positive review. (Voosen 2017) 


As one already can deduce from this short description, it seems to be an exagger- 
ation to claim that this model indeed provides an explanation in any profound 
sense. Basically, it is an ‘experimental system’ that simply highlights those ele- 
ments that play an important role in the system’s decision-making process, with- 
out actually revealing the reasoning implicit in the prediction model. Without 
access to the latter, however, it is difficult for XAI models to fulfill the great prom- 
ises of a democratic AI: To provide full accessibility, transparency and, control. 
Another interesting tool that—in a way—helped to make visible how ANNs 
work, is a now famous program called “DeepDream”, introduced by engineers and 
scientists at Google in 2015. DeepDream is a special DL-based image recognition 
algorithm, yet it operates a little bit differently than a typical CNN. First, the algo- 
rithm is trained with millions of images that show a particular object (for example, 
a cat) so that, at some point, the NN is able to predict or classify those objects (for 
example, cats) in images which it hasn’t been trained for. After the initial training, 
the network can operate in reverse. Instead of adjusting the weights of a networks 
as would be the standard procedure with the back prop algorithm, the weights 
remain unchanged, and only the input (the original image of a cat) is minimally 
adjusted. This technique has very interesting results if you apply it to images that 
do not contain any cats but are labelled as if they would. In this case, the software 
begins to modify and enhance certain patterns of images so that they start to look 
more and more like a cat, yet not similar to any particular existing one in our em- 
pirical world, but like a cat the way a neural network has learned to perceive, if not 
to say: dream it. As a result of this process, the system produces images that have 
a surreal and grotesque quality: for example, a photograph of a pizza can entail 
many little dog faces or you can also turn the Mona Lisa into a LSD-like halluci- 
natory nightmare.” The generated images reveal at least two interesting aspects: 
On the one hand, they show that DL is not an entirely mysterious technology in 
so far as the algorithm enhances familiar visual features. On the other hand, the 
images illustrate how differently the algorithm works in comparison to human 
perception, foregrounding, in other words, that it might focus on aspects of an 
image to which we usually, as humans, do not pay attention (cf. Knight 2017). But 
does DeepDream also contribute to the democratization of AI? There are certainly 


10 Fora critical perspective on DeepDream from a media-theoretical and psychoanalytical per- 
spective, cf. Apprich (2017). The following website shows some examples for the use of the Deep- 
Dream algorithm: https://deepdreamgenerator.com/#gallery 
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good reasons to be sceptical here, yet at least DeepDream offers an aesthetic ap- 
proach to illustrate the functioning of artificial neural networks and in doing so 
enabling people to reflect, for instance, on the opacity of DL technologies. And 
DeepDream has perhaps also inspired a number of artists who have been using 
DL techniques in their work for some time. For example, the Parisian artist collec- 
tive “Obvious” has explicitly declared that they want to democratize AI with their 
art. The group recently has made headlines by selling, for a considerable sum, a 
portrait at the auction house Christies which they had produced using a so-called 
GAN algorithm.” 

A third potential approach to expose the working mechanisms of a DL system 
is the so-called “Pointing and Justification (PJ-X)” model developed at the Univer- 
sity of California, Berkeley, and the Max Planck Institute for Informatics (see Park 
et al. [2016]). The model is able to justify its prediction or classification tasks by 
highlighting and documenting the evidence for the algorithmic decision using an 
attention mechanism combined with a natural language explanation. A key ele- 
ment of the system is that it is trained with two different data sets. The first one 
is meant to determine what an image shows, while the second one has the func- 
tion to reveal why something (i.e., a certain human activity or object) appears ina 
particular image. Thus, the idea is to correlate images that show objects or human 
activities not only with their description (by labelling them), but also with their 
respective explanation. For the latter purpose, each image of the training data 
is associated with three questions as well as with ten answers for each of them. 
On this basis, the system can answer questions like “Is the person in the image 
flying?” And the answer might be: “No, because the person’s feet are still standing 
on the ground” (cf. Gershgorn 2016). Again, this model—like all of the above—is 
still far away from being able to explain its own internal operations or those of 
different machine (or of another ANN, if you will). Perhaps, this specific capabil- 
ity would require that machines develop some kind of self-consciousness, or even 
a meta-consciousness. Before this happens (if this is ever going to happen), DL 
technology needs to understand reasoning, planning, causal relationships, and so 
on. For the moment, the technology of DL or ANN only provides correlations, but 
no profound causal explanations. The latter would, however, be necessary in view 
of the claims to democratize AI, if one takes into account, for example, that laws 
and other forms of regulations and organization definitely require causal expla- 
nations. However, as long as DL models cannot adaeugately address problems of 
causality, it is misleading or simply an exaggeration to speak of these approaches 
as forms of XAI. 


11 At the same time Obvious was criticized for their project because they did nothing but use a 
foreign algorithm to generate this and other of their portraits. 
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5. The Politics of OpenAl 


As lindicated earlier, providing models of an explainable and—more generally—a 
responsible AI has some obvious motivations. First and foremost, those who cur- 
rently develop DL systems have a strong economic interest to counter the social 
fears and scepticism related to a profoundly opaque AI technology. Nonetheless, 
many scientists and industrial actors underscore the political and ethical impor- 
tance of developing an explainable AI beyond the commercial aspects connect- 
ed to the interpretability problem described above. One of the most visible and 
powerful actors among those highlighting this agenda is OpenAl that started as a 
“non-profit research company” (self-description)”, also specialized on DL technol- 
ogy. Here is how the company outlined its mission goal, shortly after it has been 
founded in October 2015: 


Our goal is to advance digital intelligence in the way that is most likely to benefit 
humanity as a whole, unconstrained by a need to generate financial return. Since 
our research is free from financial obligations, we can better focus on a positive 
human impact. 


We believe Al should be an extension of individual human wills and, in the spirit 
of liberty, as broadly and evenly distributed as possible. The outcome of this ven- 
ture is uncertain and the work is difficult, but we believe the goal andthe structure 
are right. We hope this is what matters most to the best in the field. (“Introducing 
OpenAl”) 


To make sure that OpenAl is “unconstrained by a need to generate financial re- 
turn,” the founders of the company, among them, most prominently, Elon Musk 
and Sam Altman, invested more than US$1 billion in this venture. Interestingly 
enough, this initial launch posting does not explicitly or directly refer to what Elon 
Musk has named one of his key motivation for his initial investment in OpenAl, 
namely, that he regards (general) AI to be humanity’s biggest existential threat.” 
This apocalyptic view has been around since the very beginning of AI research and 
even existed before. In fact, as media scholar Bernhard Dotzler already pointed 
at the end of the 1980s, you can find most well-established projections of the fu- 
ture of Al already in the work of Alan Turing (cf. Dotzler 1989). And yet, since very 


12 Itis worth noting and also telling that OpenAl is now a “capped-profit“ company (cf. Coldewey 
2019: online). 

13 In February 2018, Musk announced that he is leaving the board of OpenAl due to a potential 
conflict of interest with his (future) work at Tesla (Vincent 2018). 
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recently, the development of AI has given us little reason to expect any dystopian 
‘Terminator’ reality to be just around the corner. 

For the first time in the history of mankind, the current situation might indeed 
be a different one vis-a-vis the undeniable fast progress of current DL technology. 
At least this is what many experts beyond Musk believe to be the case. OpenAl’s 
agenda acknowledges this new situation, but in a more nuanced, less dramatic 
manner: 


Al systems today have impressive but narrow capabilities. It seems that we'll keep 

whittling away at their constraints, and in the extreme case they will reach human 

performance on virtually every intellectual task. It’s hard to fathom how much hu- 
man-level Al could benefit society, and it’s equally hard to imagine how much it 

could damage society if built or used incorrectly. (“About OpenAl”) 


Indeed, no one is able to foresee the future of AI or can evaluate whether it will 
more likely have a positive or negative effect on society and culture. We might also 
tell ourselves that technology is never inherently good or bad, hence what matters 
only is its specific use. This argument, however, has always been a rather problem- 
atic one, since, in fact, it makes a big difference if we deal with nuclear technology 
or, say, wind power. Furthermore, even though it is rather a truism that the future 
is uncertain, we should also not forget that we can never be sure at which concrete 
historical point we might take the wrong path towards harmful applications of AI. 
It is particularly this latter argument that seems to correspond with how OpenAl 
is linking its current agenda to the problem of an unforeseeable future: 


Because of Al’s surprising history, it’s hard to predict when human-level Al might 
come within reach. When it does, it'll be important to have a leading research 
institution which can prioritize a good outcome for all over its own self-interest. 
(“About OpenAl”) 


What is interesting about this passage is the implicit assumption that the whole 
question concerning the drastic negative or positive effects of Al is still a rather 
speculative matter and not so much one that concerns the current state of tech- 
nology (“when the human-level AI might come within reach”). While OpenAl is 
right about avoiding any speculative discussion, it seems important to realize 
that DL already has both positive and problematic implications. The technology 
can do many astonishing good things, as it already has become a very powerful and 
also dangerous surveillance technology that expands the possibilities not only to 
(semi-automatically) observe the world (after being trained to do so), but to be able 
to make sense of it. 
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Very recently, it turned out that ANN/DL are not only able to identify objects, 
people, landscapes, and animals (again, after being trained to do so), but that they 
have started to understand quite complex actions and gestures. In other words: 
DL systems have begun to understand what could be called a basic form of com- 
mon-sense knowledge of the world. In order to achieve this, the ANN has been 
trained, not with photos, but with hundreds of thousands of short video-clips 
(showing certain activities and hand gestures). Hence, the specificity of media, i.e. 
here the difference between still and moving images as a mediator of DL technol- 
ogy, is an essential factor for developing advanced forms of AI. 

Ed Finn has recently argued that today’s algorithmic culture is more than 
ever driven by the “the desire to make the world effectively calculable” (2017: 26). 
Without specifically distinguishing them from learning algorithms, he regards 
algorithms in general as “cultural machines” (54) whose operations are very much 
determined by an “ideology of universal computation” (23). Indeed, one could ar- 
gue that especially modern DL technology fuels a phantasmatic version of instru- 
mental reason, precisely because it reawakens the old dream Leibnizian dream of 
a mathesis universalis, capable of perfectly understanding every aspect of our world. 
But even more, the great promise of DL is not only to make machines understand 
the world, but to make it predictable in ever so many ways: how the stock market 
develops, what people want to buy, if a person is going to die or not, and so on. 
Already at this particular moment in history, we can regard DL as the very technol- 
ogy that is capable of parsing complexities that humans aren’t able to cognitively 
process. The algorithmic power of DL lies in its potential to identify patterns by 
learning from the past to evaluate the present in order to master an uncertain 
future. And all of this happens in an ever-faster way. For example, DeepMind just 
presented a new version of its Go-program “AlphaGo Zero” that was able to learn 
the ancient board game in only three days from scratch (without implementing 
any rules how the game works or how it might be played successfully) and man- 
aged to win against the older system of 2015/16 (that beat the human world cham- 
pion Lee Sedol) by 100 to o (Perez 2017). 

The rapid speed of innovations in the field of DL should also remind us to be 
careful about quickly jumping to conclusions about what AI technology is or is not 
able to achieve. Hence, we should not only stop speculating about a distant future 
of AI, but we should also be careful about our sceptical views on what AI systems 
are capable of (or not). In general, we should acknowledge that there is still a lot of 
work for us to do if we are trying to come to terms with the current development of 
Al and machine learning technology. Maybe companies like OpenAI will succeed 
in making AI technology more accessible. But how exactly do they justify their 
central claim of democratizing AI? If we take another look at the company’s offi- 
cial website, we will realize that it provides very little information: “We publish at 
top machine learning conferences, open-source software tools for accelerating AI 


On the Media-political Dimension of Artificial Intelligence 


research, and release blog posts to communicate our research” (“About OpenAl”). 
This is basically all the company has to say about its agenda of democratizing AI, at 
least if we just consider the website's official mission statement. One thing that is 
very remarkable about this passage is the fact that there is nothing special about it. 
Facebook, Microsoft, and many other IT companies basically have the same agen- 
da (cf. Boyd 2017). 

Of course, one could argue that OpenAl at least started the current wave of 
developing a responsible and safe AI. The more important question, however, is: 
How can OpenAl legitimate its brand image and former status as a non-profit (and 
now cropped-profit) research company when it essentially does what all other big 
players in the AI game are doing—i.e. improving existing technology and/or find- 
ing the right path to develop an artificial general intelligence (AGI)? Concerning 
this matter, and very similar to the situation at DeepMind, OpenA’s research is 
focused on strategies of reinforcement learning in connection with simulations 
(like games) instead of using the common approach of supervised learning that 
depends on correctly labelled data from the empirical world (cf. Rodriguez 2017). 
Within the specific limits of their approach, both OpenAl’s and DeepMind’s agen- 
da have been quite successful. Yet, as of now, simulations are still not a suitable 
substitute for empirical learning data. If this turns out to be a permanent prob- 
lem, it will have tremendous implications for how we conceive the epistemological 
status of simulations (in many theories and histories of digital and visual culture), 
but this remains to be seen. The reason why I have highlighted this point is a dif- 
ferent one: As we just saw, there are many facets to the black box problem of DL. 
It is not my aim to get into every detail of how leading IT companies are currently 
trying to develop highly efficient AGI systems. Instead, what we can learn by tak- 
ing a closer look at those different research agendas is the simple fact that DL is 
not a homogeneous approach, but an umbrella term for very different approaches 
to AI research and design. 

Furthermore, referring to the heterogeneity of DL is not only important in 
terms of how we address the black box problem of AI, but also for how we can de- 
velop a critical perspective on intelligent machines. To provide just one example: 
A few years ago, Alexander Galloway wrote a very interesting article in which he 
politicized the black box by arguing that it is no longer a cipher like the magnetron 
technology during the Second World War, but instead has become a function that 
is more or less completely defined by its inputs and outputs (cf. Galloway 2011: 273). 
By using the term, he does not exclusively mean technical devices but refers to all 
networks and infrastructures of humans, objects, etc. that may interact with each 
other, yet thereby only articulating their external functions. Obviously, Gallo- 
way’s concept of the black box shares some similarities with how the term is used 
in the actor-network theory, though with an important difference: According 
to Galloway, the elements of a network that constitute a black box are no longer 
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able to reveal anything about themselves. In other words: He believes that those 
networks have become a black box that cannot be opened (this is also how Hilgers 
defines a black box—as system whose inner processes remain constantly inacces- 
sible; cf. Hilgers 2009). Opposed to that, for example, Michel Callon has argued 
that any black box whose actor-network operations do not adequately model the 
working of a system not only can be, but must be cracked open, thereby producing 
a “swarm of new actors” (Callon 1986). At first glance, it seems that that Galloway’s 
concept of black box could be useful to describe the infrastructures and techno- 
logical networks mediated by modern DL/ANN algorithms. But this is not as easy 
as it might seem in the first place. Galloway’s model is based on the existence of 
given inputs and outputs. Yet, ANN technology does not always operate with both 
inputs and outputs available. For example, in case of what is called unsupervised 
machine learning, the algorithm is trained without given outputs. Hence, as this 
simple example shows, if we want to understand the nuances of a DL/ML infra- 
structure as a black box, Galloway’s intervention might be of limited use. At the 
same time—and this is the aspect where the actor-network theory comes into play 
again—we cannot simply assume that the black box problem as a political (or eth- 
ical) issue only concerns the algorithm itself. Instead, the question encompasses 
many different mediators and media: legal aspects, institutional procedures, en- 
vironmental issues, existing political as well as legal regulations, and so on." 
These aspects are also important to consider if we think about how DL pro- 
grams exhibit racial or gender biases. There was great turmoil when Microsoft’s 
chatbot “Tay” was trained by Twitter users to learn racist, sexist, as well as an- 
ti-Semitic statements (Vincent 2016). This scandal has been very insightful, since 
it demonstrates how many of the operations of learning algorithms actually de- 
pend on the data and—even more importantly—on the people who label the data, 
at least in the case of supervised learning tasks. In other words: It is not or least 
not su much the algorithms that produce prejudices or political problematic out- 
come, but in fact the human actors who design and generate the learning data, 
among them the hundreds or thousands of crowd-workers hired and organized 
through platforms like Amazon Mechanical Turk or CrowdFlower. Thus, if we 
want to talk about a bias problem of AI, we should also address the general struc- 
tures of prejudices and ideology that still inform our society and thus the experts 
and workers who design the AI systems. Furthermore, this example clearly shows 


14 In order to shed light on problems of opacity concernin the broad spectrum of mediators, Bur- 
rell’s (2016) account of black boxing might be of use, with he breaks into three levels: “(1) opacity 
as intentional corporate or state secrecy, (2) opacity as technical illiteracy, and (3) an opacity 
that arises from the characteristics of machine learning algorithms and the scale required to 
apply them usefully.” 
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why it matters to take a closer look at the way certain forms of media act as key 
mediators of modern AI technology. 

Without doubt, it is rather short-sighted that the discussion on AI as a black 
box so far has focused almost exclusively on the technological aspects in the nar- 
rower sense. This also concerns the critique of a “democratic AI”. For example, phi- 
losopher Nick Bostrom recently questioned the whole logic of making AI safer by 
making it more transparent: “If you have a button that could do bad things to the 
world, you don’t want to give it to everyone” (quoted after: Metz 2016). Prima facie, 
this argument may sound convincing, but at the same time, it seems a little bit 
odd. If we think about nuclear weapons, for example, one can easily observe how 
complicated it is to keep a possibly “dangerous button” just for yourself. (We might 
also point to recent discussions here about the US president’s right to decide ifhe 
uses nuclear weapons as a first strike or not). I do not want to argue that the con- 
cept of a balance of deterrence during the Cold War actually had a peace-securing 
effect, nor do I want to put the specific technology of nuclear weapons on the same 
level as AI. I just want to illustrate why the whole practice and discourse of a re- 
sponsible or transparent AI may be more complicated than Bostrom’s statement 
suggests. Neither is it true that the only alternative to the idea of a transparent 
AI would be to keep all the relevant knowledge about Al secret. At least, the latter 
strategy cannot be an option for OpenAl, since it would destroy the company’s 
very identity. 

Furthermore, it is important to highlight that as much as the black box prob- 
lem does not only concern the technology itself, we also have to acknowledge that 
any attempt to democratize AI cannot just be reduced to activities of open-sourc- 
ing the tools and knowledge around its technology (cf. for a further critical view 
on AI asa black box beyond issues of transparency and accountability: Matzner 
2017). It is not a dystopian position to argue that we already live in a post-priva- 
cy age where people have very little control over the processes of data collection, 
storage, processing, and transmission related to their personal lives and activi- 
ties. The revelations of Edward Snowden have already confirmed the worst con- 
spiracy theories about surveillance to be true (Sprenger 2015). The problem here 
is not only that companies or secret services, or governments in general, collect 
and analyse private data against our will. All too often, many people simply do not 
care enough about the data that they generate and circulate while being online 
or using this or that application. And even if they individually try to protect their 
private information, there is no guarantee that their friends, family, or colleagues 
will do the same.” These aspects have been the subject of cultural critique long 
before the current AI boom took off. We should therefore not simply discuss how 


15 For example, as it has been revealed during the Cambridge Analytica scandal, the company 
used a back door in the Facebook API from c. 2012-2015 where a Facebook friend’s choice to opt 
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to democratize AI but continue our efforts to secure democratic standards for our 
data-driven world in general. To achieve this goal, linking the political analysis of 
AI with a broader discussion about datafication is nothing more than a first step, 
but arguably a very important one. 

Currently, it is hard to think of any institution or law, globally or locally, that can 
protect us from the dangers of AI as well as the misuse of big data. Neither do we 
have any profound reason to believe that specifically companies like Open-Al, Face- 
book, or Google will achieve this goal. At the same time, it is perhaps short-sighted 
to think of these tech companies as the enemies of a democratic digital culture just 
because they are the hegemonic forces that control both the data as well as the intel- 
ligent algorithms capable of making sense of it. Obviously, there are dangers of AI 
that are more urgent, for example, if non-democratic states use AI or DL technology 
to oppress political opposition or perhaps terrorists using AI for cyberattacks. This 
threat is not an instance of a big data or AI paranoia: As experts have recently demon- 
strated, by only having access to a so-called API, you are able to reverse-engineer ma- 
chine learning algorithms with close to 100% accuracy. Hackers are able to steal Al 
technology from IT companies like IBM or Microsoft for whatever their specific goals 
might be (for the technical details, see Claburn 2016). Of course, having a truly open 
AI might solve this particular problem in the first place. But then again, one has to 
ask how we can make sure that an open Al is not used for harmful purposes. 

As of now, OpenAl seems less concerned with any concrete political vision of 
Aland more keen on participating in the competitive race towards developing an 
artificial general intelligence. Hence, it is quite seductive to believe OpenAl’s po- 
litical or ethical agenda is basically a PR stunt and nothing else. But instead of 
simply questioning if OpenAl’s concrete practices matches their agenda or not, 
it might be more productive for a media-political account to discuss the politi- 
cal implications and effects of a transparent or responsible AI in the context of a 
broader focus: How the technology of learning algorithms reshapes the conditions 
of an instrumental rationality so deeply connected with every aspect of our digital 
culture and society. And this important project just has started. 
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1. Introduction 


Artificial intelligence is a discipline within computer science that deals with the 
development of software-based systems that provide functions which require the 
execution of what is typically called (human) intelligence. However, since there is 
no widely accepted definition of human intelligence, there is also no widely ac- 
cepted for artificialintelligence, sometimes also called machine intelligence (Legg, 
2007). Al uses methods and tools from logic, probability theory, and continuous 
mathematics in order to provide perception, reasoning, learning, and action via 
software-based systems (Russell, 2016). And it provides already numerous prac- 
tical applications in transportation, energy supply, health services, finance and 
banking as well as law and regulation: “AI technologies already pervade our lives. 
As they become a central force in society, the field is shifting from simply building 
systems that are intelligent to building intelligent systems that are human-aware 
and trustworthy.” (Stone, 2016) 


Si 
Speech 
Identification 


li 
Image 
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Sr Da Te 
General e Text 
Identification J Pr? Analytics Extraction 


Fig. 1: Functional components in AI by Hammond (2016): Recognition of speech (Sr), 
audio (Ar), face (Fr) and image (Ir) and general recognition (Gr), Identification of speech 
(Si), audio (Ai), face (Fi) and image (Ii) and general identification (Gi); Data analytics 
(Da) and Text extraction (Te); Predictive inference (Pi), Planning (Pl), Explanatory 
inference (Ei), Problem solving (Ps), Synthetic reasoning (Sr), and Decision making 


246 


Ina Schieferdecker/Jürgen GroBmann/Martin A. Schneider 


(Dm); Language generation (Lg) and understanding (Lu); Relationship learning (Rl), 
Category learning (Cl) and Knowledge refinement (Kr); Mobility at large (Ml) and at 
small (Ms); Manipulation (Ma), Communication (Cm) and Control (Cn), which can be 
used standalone or in combination e.g. to predict future events by recognizing sounds 
of technical systems and/or identifying images representing system states and/or 
correlating data and recognizing specific facts. 


Technologies that are used to build AI by machine learning (in short ML), which is 
about improving problem solving accuracy or efficiency by learning to do some- 
thing better, are numerous. Machine learning can e.g. be grouped along the learn- 
ing type into methods for supervised, unsupervised or semi-supervised learning 
or along the knowledge extraction by symbolic computation or sub-symbolic pro- 
cessing. They can also be grouped along the principal approach, e.g. into regres- 
sion, instance-based, regularization, decision tree, Bayesian, clustering, neural 
network, deep learning, and quite many other algorithms. Based on these, like- 
wise numerous AI applications can be developed. Hammond (2016) presented a 
first taxonomy of AI functional components (Fig. 1). No matter which functional 
components are being used, AI-based systems are realized by use of software or 
also by use of sensors and actuators for the interconnection with the environment 
(Fig. 2). The software uses data which are interpreted by algorithms in order to 
provide automatisms for parts of or for entire processes in technical systems like 
in car engine control or in socio-technical systems like in autonomous driving. 


« Internet of Things e Measurements 

« Digital and « Real-Time Data 
Digitalised e Data Series 
Infrastructures Sensors « Data Streams 


* End Devices Actuators 


e Decision e Computability 
Sovereignty e Correctness 

+ Traceability e Data Quality 

e Fairness 


Fig. 2: Elements of software-based systems (WBGU, 2019). Sensors are part of the 
Internet of Things and generate different kinds of data such as measurements, series 

of measurements or data streams. Algorithms use these data in their computations or 
as training data. The algorithms are constrained by complexity, computability, and 
performance limits and possibly by the (in-)correctness of the implemented computation 
logic and by the (un-)biased (training) data. In result, software-based systems offer 
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automatisms for which it is essential to agree (and assure) decision sovereignty, 
traceability and fairness. Any decision in respect to the environment can finally be fed 
via software (into the cyberspace) and via actuators (into the environment). 


2. Software Verification and Validation 


Since any Al is also a software-based system, it is to be seen to which extent AI can 
be verified and validated with the established verification and validation (in short 
V&V) methods for software in general. V&V methods for software were revealed 
already with the software crisis back in 1968 (Wirth, 2008), when the term soft- 
ware engineering was coined. It pointed at the difficulties to design and develop 
useful and trustworthy software with the given resources and within the given 
time: “The major cause of the software crisis is that the machines have become 
several orders of magnitude more powerful! ...(A)s long as there were no ma- 
chines, programming was no problem at all; when we had a few weak comput- 
ers, programming became a mild problem, and now we have gigantic computers, 
programming has become an equally gigantic problem.” (Dijkstra, 1972). And the 
newly coined term pointed at the necessity to develop practical and scalable engi- 
neering methods for software development. Since then, constructive and analyt- 
ic methods for software quality engineering have been developed. They include 
methods for software engineering processes, software engineering tools and for 
software as such. A rough overview on these methods is given in Fig. 3. 


Software quality engineering 


Constructive methods Analytic methods 


Model-driven engineering 
Prototyping 


Software architectures, patterns Piloting Testing 
and anti-patterns S 5 
: Simulation 
Program generation and 
wna: Formal proof 
Programming languages and Monitoring and watch-dogs 
guidelines V&V frameworks and tool 
Frameworks and tool boxes boxes 


Fig. 3: Overview on software quality engineering methods. Software quality begins 
with the software design that is represented by software architectures which can make 
use of software patterns. Programs can be (partially) generated from these software 
designs and/or refined. The programs use typically high-level programming languages 
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which offer guidelines for best practice programming and which are supported by 
programming frameworks and tools. The achieved software quality is typically tested, 
checked by simulation or proven formally. The running software can be monitored and 
watch-dogs can check for constraint violations at run-time. All these analytic methods 
can also be automated by V&V frameworks and tools. Three specific (sets) of methods 
can be used both constructively and analytically: that is the use of model in software 
engineering, the early prototyping of software (or of V&V software) and the piloting of 
software (or of V&V solutions). 


The software (program or code) tells the computer what to do, “but that may be 
much different from what you had in mind”. (Joseph Weizenbaum, Computer Scien- 
tist, 1923-2008). However, by the systematic use of software quality engineering 
methods, software can be developed such that it is safe, secure, and trustworthy 
and that it can analyze and compute more data than any person and can do this 
more reliably. 

Numerous international software engineering standards put the ground for 
software quality such as ISO/IEC 25010 (ISO, 2011) for software quality require- 
ments and evaluation (SQuaRE) and software quality models. It argues about 
quality in use, external quality and internal quality of software and differentiates 
between functional suitability, reliability, usability, security, compatibility, porta- 
bility, maintainability and performance/efficiency. 

While these are all important software quality aspects that evolved over de- 
cades, interestingly, new aspects arise for AI in their use within socio-technical 
systems. Apparently, 


e understandability, i.e. users and operators can get to know the features and 
services of the systems, 

e interpretability, i.e. users and concerned people have access to clarifications of 
outcomes and their potential impacts, 

e traceability, i.e. users and concerned people have access to more detailed anal- 
ysis of outcomes in relation to a given situation/problem statement, 

e explainability, i.e. users and concerned people receive descriptions, reasoning 
and justifications on the outcomes, as well as 

e fairness, i.e. concerned people are treated the same wrt. commonly agreed 
rules for treatment, gain much more momentum. 
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3. Al Verification and Validation 


Indeed, AI requires to quite some extent additional methods and tools for V&V 
(Van Wesel, 2017) since well-established testing technologies are short in V&V of 
AI. This is not only true because of the additional socio-technical quality aspects 
(see above), but also due to the different nature of logic-based software (most of 
the software in general so far and some of AI) and statistics-based software (most 
of AI, in particular in machine learning). Testing has limitations with respect to 
the dynamics of ML, the sheer size of the problem domain and the underlying 
oracle problem (Xie, 2011). 

In addition, most of the AI is controlled by data. In this sense, a neural net- 
work is a generic function approximator whose structure reflects the actual func- 
tionality only to avery small extent. Hence, source code-oriented V&V techniques 
such as static analysis or white-box tests are only of limited use in this context. 
On the other hand, the trustworthiness and quality of the data becomes a central 
issue for the overall quality of the systems. 

However, since systematic dynamic testing of software is the best-known 
and most effective V&V method, it will most probably also form the main basis 
for testing ML. In recent decades, research has developed industrial-grade tech- 
niques for increasing the quality, efficiency and reliability of testing. This includes 
in particular, automation strategies for dynamic testing such as automating test 
executions with test technologies like TTCN-3 (Testing and Test Control Notation 
[Grabowski, 2003]), for model-based testing to automate the generation of tests 
(MBT [Utting, 2012]), as well as the use of search and optimization algorithms 
for automated test selection and test suite reduction (Harman, 2015). Moreover, 
the combination of dynamic testing with verification approaches like source code 
analysis, model checking and symbolic execution allows for improvements in 
testing, that combines the rigor of verification processes with the scalability of dy- 
namic testing (Godefroid, 2018). These techniques are applied to testing for func- 
tional as well as extra-functional properties like performance or security (Schiefer- 
decker, 2012). Finally, the close integration of testing with system development 
processes and risk management (Felderer, 2014) improved the efficiency and 
transparency of testing so that testing has matured as one of the most important 
software quality measures in industry. Still, test automation as well as the use 
of models in testing are still underexplored: although a strong test automation 
is required, less than 14% of software testing professionals say that they use MBT 
(Binder, 2015). The potential of risk-based testing to steer test processes based on 
uncertainties has been shown especially in the area of critical system in terms of 
security and safety, which will likewise be applicable to AI (Erdogan, 2014). 
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Quality, traceability, explainability, 
understandability, fairness 


Unbiased data 


Formal verification Dynamic testing 


Predeployment and runtime V&V of AI 


Fig. 4: The AI VSV pyramid. AI-based systems are to be verified and validated both in 
predeployment phases and at runtime. A combination of V&V methods from formal 
verification and dynamic testing is recommended, in particular for safety- and security 
critical AI-based systems. V&V will help to assure both quality and explainability 
requirements as well as enable the justification of bias in the (training) data used in Al. 


Research on dedicated methods for verification and validation of ML is still at its 
beginning. Even so, testing is already part of the overall training set-up in ML, 
most testing is done to achieve more accurate models with respect to the initial 
training objectives. In supervised learning for example, test and validation data 
sets are used to provide evaluation of the ML model. Validation data sets are typ- 
ically used during training to fine-tune the model parameters while test data sets 
are used on the final model to measure generalization errors. However, since in- 
dividual test sets only provide a single evaluation of the model and have limited 
ability to characterize the uncertainty in the results, more advanced statistical 
testing approaches like cross-validation are used for model selection. 

Ghosh et al (2016) combine ML and model checking in such a way that if the 
desired logical properties are not satisfied by a trained model, the model (‘model 
repair’) or the data from which the model is learned is modified systematically 
(‘data repair’). Fulton and Platzer (2018) propose to combine formal verification 
with verified runtime monitoring so that safe learning can be guaranteed. The ap- 
proach intervenes in the learning process whenever safety properties are violated 
and guides the learning process so that the result is compliant with the verifica- 
tion model. Approaches like DeepXplore (Pei, 2017), DLFuzz (Guo, 2018) and Ten- 
sorFuzz (Odena, 2018) provide metrics for the quantification of neural coverage 
and simplify test automation. DeepTest (Tian, 2018) enables systematic testing of 
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neural networks under realistically changing environmental conditions especially 
for use in the automotive domain. 

One of the socio-technical limitations of ML is the lack of transparency, i.e. its 
black box-approach. In order to address it, different approaches have been pro- 
posed such as 


e modelinterpretation for image classifications, e.g. by understanding the acti- 
vation maximization with saliency maps (Simonyan, 2013), 

e model explanation by sensitivity analysis and local explanation vectors to pro- 
vide reasons for the decisions of any classification method (Baehrens, 2010), 

« model decomposition for interpreting generic multilayer neural networks by 
decomposing the network classification decision into contributions of its in- 
put elements (Montavon, 2017), 

e extraction of decision trees from input data generated from trained neuronal 
networks (Krishnan, 1999), 

e relevance propagation by pixel-wise decomposition of non-linear classifiers 
(Bach, 2015), and 

e deconvolution methods to give insight into the function of intermediate fea- 
ture layers and the operation of classifiers (Zeiler, 2014). 


Another well-established way is to use test scenarios, i.e. test cases and their test 
data, for explaining ML decisions. The other socio-technical limitation of ML is 
the potential lack of fairness, i.e. the potential bias. Here, systematic generation 
of (training) data that cover well required categories and properties as known 
from test data generation is of help (Nguyen, 2016). 

The ability to effectively test AI will be fundamental for the acceptance in 
broad scale and central for safety-critical areas like transportation and automo- 
tive, healthcare, or industrial automation. The provisioning of test technologies, 
tools, test scenarios with test cases and test data for AI will not only be a solid 
basis for V&V but also help in explaining AI and making them more transparent 
and unbiased. They can also be used to ensure safety and security of AI during 
runtime. 

And last but not least, the tools for safeguarding AI contribute also to the de- 
mocratization of AI: They are the basis for confirming or witnessing outcomes 
whenever AI-based systems are to be accounted. They can also become a digital 
common for the comparison and benchmarking of AI-based systems and by that 
contribute to a shared knowledge basis of AI. 
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Al, Democracy and the Law 


Christian Djeffal 


Digital technologies are in the process of reconfiguring our democracy. While we 
look for orientation and guidance in this process, the relationship between tech- 
nology and democracy is unclear and seems to be in flux. Are technology and de- 
mocracy mirroring each other?! The internet was first hailed as genuinely demo- 
cratic technology and ultimate enabler of democracy. It is now often perceived as 
a major threat to democracy. The story of artificial intelligence (AI) might turn out 
to be quite the opposite. While there are many reflections on AI as a threat to or 
even as the end of democracy,” some voices highlight the democratic potentials of 
AI.’ As is often the case, the research results depend on the premises underlying 
the research. This chapter is based on the assertion that technologies and media 
shape human affairs to a large extent, but that technology in turn is also shaped 
by human choices and decisions. There is a huge potential to endanger, game or 
even abolish democratic processes. On the contrary, there might also be opportu- 
nities to further democracy. Therefore, the extent to which AI impacts democracy 
is subject to the paths that are chosen in research, development and application 
of Al in society. 

The main purpose of this chapter is to highlight the room for choice in the con- 
struction of AI and its impacts on the future of democracy. It will also inquire 
into how law and jurisprudence relate to these questions. From this perspective, 
current impacts of AI on democracy have an important indicative function. But in 
the face of further possibilities of inventions and regulative measures on different 
levels, they are only precursors to what will and should be possible. In that sense, 
this chapter is also an attempt to deal with developments and inventions we can- 
not yet grasp. The main argument is that it might be possible to influence them 
nevertheless. Therefore, the chapter will reflect on the possibility and necessity to 
democratize AI from a legal and jurisprudential perspective. It will then look at 
different ways to democratize Al. 


1 On this question see Hofmann (2018). 
2 Hofstetter (2016); O’Neil (2016). 
3 Helbing (2019); Ennals (1987). 
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I. Democratizing Al: Possibility and Necessity 
A. Understanding the Openness and the Power of Artificial Intelligence 


In order to understand the relationship between artificial intelligence and democ- 
racy, it is necessary to clarify the concept of AI. The concept’s crucial aspect lies 

not in a clear-cut definition of AI but in its openness. AI is a very broad concept in- 
deed, and this might be the reason why this concept has outperformed other ideas, 
such as cybernetics, and is today the general term used in science, politics and the 

economy. Artificial intelligence is a term denoting a research question that in- 
spires a hole sub-discipline of computer science today. This research question has 

been summarized as follows: Can systems solve complex problems independent- 
ly?* The openness can already be seen in the initial definition of the term from 1955 

in a grant proposal to the Rockefeller Foundation: 


We propose that a 2-month, 10 man study of artificial intelligence be carried out 
during the summer of 1956 at Dartmouth College in Hanover, New Hampshire. The 
study is to proceed on the basis of the conjecture that every aspect of learning or 
any other feature of intelligence can in principle be so precisely described that a 
machine can be made to simulate it. An attempt will be made to find how to make 
machines use language, form abstractions and concepts, solve kinds of problems 
now reserved for humans, and improve themselves.° 


It is clever to frame a grant proposal in a way that encourages the imagination 
of those reading it. The way AI was used here does exactly that. The first aspect 
regarding the openness of AI that can be derived from this definition is that Al 
is a research question. It is not a theory offering explanations. It is not a general 
hypothesis or an idea framing certain aspects in a particular manner. The general 
research question of whether systems can solve complex problems independently 
is based on certain conjectures, but those are reduced to a minimum. The fact that 
Alis a question might also explain the longevity of the term. AI has seen so many 
ups and downs that commentators speak about “AI winters” and “AI summers.” 
As long as the general research question underlying Al is not solved in a manner 
that cannot be improved, it will continue to be interesting for Al researchers. An- 
other aspect of the openness of AI relates to its basic assumptions. Comments by 
John McCarthy, one of the grant applicants and important figures in AI research, 
suggest that the term AI was coined to avoid the assumptions made in cybernet- 


4 Mainzer (2019: 3). 
5 McCarthy/Minsky/Shannon (1955). 
6 Sudmann (57). 
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ics research and to get around the influence of Norbert Wiener.” While Wiener 
certainly made great contributions to the field of computer science and touched 
upon many important questions of AI that are still relevant today, he did so from 
another perspective. His idea of cybernetics, commonly held among many other 
important colleagues, is a general theory with strong assumptions. In contrast, 
the term AI has traditionally accommodated quite different views. One general 
disagreement has been termed as the strong and weak AI hypothesis.? The strong 
AI thesis departed from the idea that AI can either replicate or even surpass hu- 
man intelligence. In contrast, the weak AI thesis only requires machines to act as 
if they were intelligent. It focusses generally on certain specific problems to be 
solved. 

Another aspect of the openness of Al is that it does not relate to a single tech- 
nology but to a whole set of technologies.’ At the moment, technologies of machine 
learning! are considered to be either state of the art or even “real AI.” Artificial 
neuronal networks, for example, fulfil certain tasks such as image recognition. 
They are trained on the basis of a great amount of training data, which is labelled 
so that the mathematical models underpinning the learning may continuously 
be adapted and improved. In contrast, generative adversarial networks improve 
themselves in an adversarial manner without the input of human training data. 
There are still many general ideas and architectures that might have been more 
popular in the past, such as decision trees, or that might become more popular 
in the future, such as evolutionary AI. Since AI is open for new approaches and 
breakthroughs, AI research continues to be a moving target. Systems that rep- 
resented state-of-the-art-Al at one point in time do not qualify as being truly 
intelligent later. Different technologies require different resources. While AI is 
sometimes associated with big data applications that rely on training or analysis 
of huge amounts of data, big data is not a necessary requirement. There are also 
small data applications or applications that do not require significant training 
data at all. The resources vary accordingly. Artificial neural networks need large 
amounts of training data, sufficient memory space to store this data and enough 
power to compute it. It is especially important to note that the training data has 
to be annotated by human beings. Whether it is the interpretation of x-rays, skin 
cancer detection or crosswalk recognition in the context of automated driving, 
the data to train deep neural networks is dependent on human input. Large pools 
of human resources were even more crucial with the old expert systems popular 


7 McCarthy (1989). 
8 Fora discussion see Russell/Norvig/Kirchner (2012: 1020). 
9 Gasser/Almeida (2017: 59). 


10 For overviews from different perspectives see Shalev-Shwartz/Ben-David (2014); Sudmann/En- 
gemann Goodfellow/Bengio/Courville (2016). 
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in the 1990s. Experts had to design decision trees in many cases, which would then 
assist other people. 

Furthermore, the general purposes of AI are also open. While it is often as- 
sumed that AI is synonymous with automation, there is indeed a disagreement 
about whether the goal of AI is rather augmentation than automation. While 
automation relates to the replacement of humans by machines, augmentation 
focusses on human-machine interaction in order to amplify human capabilities. 
This augmentation paradigm proved to be influential in different areas of com- 
puter science. Even the earliest research agenda by its most influential proponent 
Douglas Engelbart shows that there is a clear link to the research agenda of arti- 
ficial intelligence: 


By “augmenting human intellect” we mean increasing the capability of a man to 

approach a complex problem situation, to gain comprehension to suit his particu- 
lar needs, and to derive solutions to problems. Increased capability in this respect 

is taken to mean a mixture of the following: more-rapid comprehension, better 

comprehension, the possibility of gaining a useful degree of comprehension in 

a situation that previously was too complex, speedier solutions, better solutions, 
and the possibility of finding solutions to problems that before seemed insoluble. 
And by “complex situations” we include the professional problems of diplomats, 
executives, social scientists, life scientists, physical scientists, attorneys, design- 
ers—whether the problem situation exists for twenty minutes or twenty years. 
We do not speak of isolated clever tricks that help in particular situations. We re- 
fer to a way of life in an integrated domain where hunches, cut-and-try, intangi- 
bles, and the human “feel fora situation” usefully co-exist with powerful concepts, 
streamlined terminology and notation, sophisticated methods, and high-pow- 


ered electronic aids.” 


The systems capable of such an automation are to be “sophisticated” and able to 
deal with complexity. Those systems are to be combined with human intelligence. 
They are not intended to replace it. So, the general aim of artificial intelligence is 
also open regarding augmentation and automation. This openness in the general 
aim in the relationship of AI and humans is reflected in the variety of purposes 
for which AI systems can be used. The set of technologies described by the term 
Al are so called general-purpose technologies (GPTs). While the concept of GPTs 
has mainly been applied in economics,” it fits well as a category for analyzing so- 
cial impacts of technology. The many purposes for the use of coal and steel have 
been captured in the phrase “swords to ploughshares.” Maybe the same might 


11 Engelbart (1963: 1). 


12 Rousseau (2009). 
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be said about AI, which can fuel lethal autonomous weapon systems as well as 
assistive care robots. In order to understand AI, the comprehension of the gen- 
eral-purpose nature of the respective technologies is of the utmost importance. 
The technologies comprising AI are neither exclusively tied to certain risks and 
challenges nor to certain opportunities and advantages. There are many counter- 
intuitive examples for this proposition, but data protection and privacy are again 
very illustrative in that regard. AI systems can certainly be a threat to privacy and 
data protection as they allow the extraction of a lot of personal information. One 
interesting aspect is Al-powered shadow profiling. This means that people are 
profiled without any significant activity of their own. The information is provid- 
ed by people around them. Circumstantial evidence, such as search queries from 
other persons in a social network, allow smart systems to reconstruct a profile of 
a person within that network and collect relevant personal information, without 
them having personally revealed anything. However, AI can also help to serve as a 
privacy enhancing technology. There is, for example, a general push for chatbots 
that learn the privacy preferences of a person in a short and simple conversation 
and then go on to adapt the privacy preferences in all networks and online-ser- 
vices the person uses. The purposes of AI systems can, therefore, both enhance 
and threaten privacy. As will be shown, the same is true for other principles and 
values such as democracy. 


Aspect of Openness Alternatives 
Research question Weak Al thesis, strong Al thesis 
Technologies Machine learning technologies (artificial neural networks, genera- 


tive adversarial networks), good old-fashioned Al 


Resources Data, common sense, computation... 
Aims of use Automation vs. augmentation 
Purposes General purpose technologies: can go many ways regarding 


purposes like transparency and data protection 


Table 1: Dimensions of Openness of AI 


The importance of this openness can be appreciated to a fuller extent when recog- 
nizing the ways in which democracy can be shaped by technology. Firstly, there 
are different understandings and constructions of the meaning of democracy. 
While there is acommon thread of self-government of a people, there are differing 
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views on how this self-government is to be exercised. Democracy is constituted in 
actual practices in society. Technology has always played a huge part in the actual 
practice of democracy. Democracy and technology are intertwined. “Democra- 
cy is not enacted and then mediated. It is preformed through acts of mediation. 
Technologies of mediation are and always have been inherent in the social enact- 
ment of democracy.”” One, therefore, can go as far as tying practices of the use of 
certain technologies to specific ideas of democracy.” The use of technologies con- 
figures democracy. In the case of AI, being a set of general-purpose technologies, 
this configuration is generally open. Democracy is a process rather than a fixed 
and attainable state. It has to be constantly realized, using means like technolog- 
ical innovation, institutions, markets and competition, law and administration.” 
In the face of this openness, it is interesting to look at current and potential uses 
of Al in the context of democracy. 


B. Empirical Insights 


While general purpose technologies like the internet or AI can play out very differ- 
ently, they are usually described in a particular way. The discourse on the internet 
and democracy began by hailing the potential beneficial effects of the internet 
on democracy.” Regarding Al, it seems to be the other way around: it is mainly 
regarded as a threat to democracy. Al is seen to have the potential to obstruct es- 
tablished democratic processes like elections and votes. There is also a fear that AI 
takes over decision making in many contexts. In order to paint a more nuanced 
picture, one has to appreciate the contingency of the technology and how it can 
be used in very different ways. The literature on the internet today recognizes 
its positive and negative effects on democracy.” The contingency of the internet 
means that “like every medium before it, from the alphabet to television, [it] is 
shaped by the ways that society chooses to use its available tools.”"® 

The general-purpose nature of AI is also reflected in its relationship with 
the democratic process, especially in the context of elections. In this regard, AI 
is generally perceived as a threat. There have been several attempts to influence 
elections through automated systems that preformed different tasks. Fake news 
are spread in the context of elections to block and obstruct political discourse and 
to target voters on a granular level in order to engage or disengage them from 


13 Coleman (2017: 27). 

14 Bozdag/van den Hoven (2015). 

15 Irrgang (2002: 173). 

16 Pernice (2016). 

17 Foran overview see Ceron/Curini/lacus (2017: 6). 


18 Coleman (2017: vii). 
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voting.” One of the activities that has been fueled by AI and other digital tech- 
nologies is micro-targeting. Micro-targeting denotes attempts to influence the 
behavior of people based on personal profiles and actions that are grounded in 
specific features of that profile. Those profiles provide specific information about 
certain persons; people can then be targeted individually through social media 
advertising instead of being addressed as part of a group with political posters or 
TV commercials. These actions can range from attempts to influence or obstruct 
democratic discourse to influencing or obstructing the actual decision-making 
of individuals. While the first micro-targeting efforts were used for canvassing 
campaigns, in which humans went door-to-door in order to influence the elector- 
ate, AI can also play a role in actions based on granular profiles of certain people. 
There have been several reports about the use of such technologies. Whereas the 
elections in the United States and Brazil and the Brexit vote have made the news, 
their use has also been debated in states like Switzerland and Austria.” AI sys- 
tems can enhance the possibilities of micro-targeting on different levels. AI can 
help with the extraction of information by crawling the web and analyzing oth- 
er sources of unstructured data. AI systems can also help to profile people. Fur- 
thermore, AI systems can automatically address persons based on their profiles 
through different channels like social media. Several aspects of these campaigns 
using micro-targeting are problematic.” First, the respective data has often been 
collected from public sources, in some instances illegally. This violates the respec- 
tive persons right to data protection if the data was collected and used illegally. It 
also violates their right to personal autonomy, in that they are being influenced 
based on the collected data. Opting out of micro-targeting is not yet an option. 
What is more, micro-targeting can also be used for purposes of manipulation. Re- 
search on the topic also mentions the possible beneficial impacts—such as ensur- 
ing that voters receive the information that is relevant for them.” This could also 
make specific topics more relevant for elections and enhance the importance of 
certain groups, particularly when they are spread out and not organized.” There- 
fore, AI could help those conceived to be weak and less powerful to obtain more 
and better information.” 


19 Bodö/Helberger/Vreese (2017: 3). 
20 Eidgenössischer Datenschutzbeauftragter/Konferenz der schweizerischen Datenschutzbeauf- 
tragten (2018); Der Standard (2019). 


21 A mapping of the threats can be found with Zuiderveen Borgesius/Möller/Kruikemeier/Ö 
Fathaigh/Irion/Dobber/Bodo/Vreese (2018: 87) On the same page, they collected reference on 
privacy and manipulation trends. 


22 Zuiderveen Borgesius/Möller/Kruikemeier/Ö Fathaigh/Irion/Dobber/Bodo/Vreese (2018: 84ff). 
23 Ibid. 
24 Ennals (1987: 14). 
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This shows again the general-purpose nature of AI and the difficulty of putting 
it into one box. Micro-targeting can be detrimental, but it can also be beneficial 
to democracy. Yet, the applications existing today are only a preliminary view of 
what could be possible. Technological improvements, but more importantly also 
creative and innovative uses of the technology could lead to an even more pro- 
found impact of AI solutions on democracy. AI solutions can be something genu- 
inely new or turn existing possibilities upside down. One example would be to em- 
power voters through targeting and profiling candidates. A smart search engine 
could help to identify information concerning how parties or candidates think 
about certain issues. Empowering voters even further, one could come up with 
Al systems that predict future government behavior. One could try to compute the 
probability that parties or candidates act on certain promises. Indeed, it seems 
to be not entirely impossible to predict the likelihood of the question of whether 
certain promises will be kept in the future. This would be a use of profiling in a 
completely different way. While such a profiling of candidates and parties raises 
a series of problems and issues, it shows that the use of AI can vary greatly and 
also support voter empowerment. It could open their decision-making potential 
as opposed to narrowing it. Whereas there is currently great concern for using Al 
in the context of elections and votes, the future impact of Al is in fact open. 


C. Law and Technology: Limitation, Motivation, Design 


The law and technology have a multi-facetted relationship. This relationship can 
be broadly summarized in three functions: limitation, motivation and design. The 
impact of law on the relationship of technology and democracy will be explained 
along these lines. The law can add to the democratization of artificial intelligence 
in different respects. To include all these functions in the picture is particularly 
important since they highlight different perspectives that are best suited to create 
a full picture of the challenges and opportunities of Al in relation to democracy. 


1. Legal Limits and Democracy 

Human rights limit the use of AI, especially by public authorities. Human rights 
also trigger the need for democratic justification. Thereby, they further limit the 
possible uses of AI. The function of the law as limit to technology is possibly its 
best-known function. Legal obligations stemming from data protection, for ex- 
ample, limit the use of technology in several respects. Data protection law can 
ban the use of training data in machine learning, because there may be no legal 
grounds for such use or existing allowances do not cover the respective purpose. 
For instance, under the EU General Data Protection Regulation (GDPR) data pro- 
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cessing has to be justified according to Art. 6.” Systems taking automated de- 
cisions have to comply with Art. 22 GDPR. This provision allows such decisions 
only when the requirements in sections 2 and 3 are met.” Sections 2 and 3 refer to 
decisions based on contracts, statues or explicit consent. 


2. Motivation 

The law can also motivate the use of technology in different forms. This motivation 
can relate to “the development, advancement and application of technology by the 
administration or even make it compulsory.” There are different ways in which 
democracy as a legal principle can motivate the use of technology and AI specif- 
ically. Looking into international instruments about democracy, one can spot 
questions of technology in different contexts.” In human rights law, there are 
several rights that point to democratic governance. Some human rights instru- 
ments explicitly point to the crucial importance of technology in order to enhance 
democracy.” One area in which this is of particular importance is the inclusion of 
persons who are not yet able to effectively participate in democratic procedures 
and democratic discourses. The United Nations Convention on the Rights of Per- 
sons with Disabilities. Art. 4 para. 1 (g) obliges “to undertake or promote research 
and development of, and to promote the availability and use of new technologies, 
including information and communications technologies, mobility aids, devices 
and assistive technologies, suitable for persons with disabilities, giving priority 
to technologies at an affordable cost.” This is an example of a progressive human 
rights clause that motivates states and other actors to employ technologies in or- 
der to further human rights. Many AI technologies help persons with disabilities, 
especially blind and death people. These technologies also empower their respec- 
tive users to participate in democratic discourse. Therefore, Art. 4 para1(g) has an 
effect on people’s democratic inclusion. 


3. Design 

Another function of the law is to structure and guide the design process. The law 
sets design goals, it shows how to balance different goals and even highlights pos- 
sibilities to solve issues on the technical level. A good example for that is the priva- 
cy by design clause in Art. 25 sec. 1, which provides as follows: 


25 Art. 6 provides that processing of data is only lawful if its requirements are met. 
26 Abel (2018); Martini (2018). 

27 Djeffal (2019: para 16). 

28 This research is based upon the collection of documents by Ehm/Walter (2015). 


29 See for example ga-Res. 68/164. Strengthening the Role of the United Nations in Enhancing 
Periodic and Genuine Elections and the Promotion of Democratization, United Nations A/ 
RES/68/164, adopted by the General Assembly on 18 December 2013 (7oth plenary meeting). 
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Taking into account the state of the art, the cost of implementation and the nature, 
scope, context and purposes of processing as well as the risks of varying likelihood 

and severity for rights and freedoms of natural persons posed by the processing, 
the controller shall, both at the time of the determination of the means for pro- 
cessing and at the time of the processing itself, implement appropriate technical 

and organisational measures, such as pseudonymisation, which are designed to 

implement data-protection principles, such as data minimisation, in an effective 

manner and to integrate the necessary safeguards into the processing in order to 

meet the requirements of this Regulation and protect the rights of data subjects. 


Art. 25 section 1 entails a direct obligation to include privacy considerations into 
the process of designing or adopting an application. It is, however, also possible to 
have rather indirect obligations. It has recently been claimed that constitutional 
principles such as human rights, the rule of law and democracy also should be 
included in the process of designing AI.*° This would further the law’s function to 
influence technologies at a very early stage. These obligations also have to be ap- 
plied by those developing the systems directly. In order to meet those obligations, 
several methodologies have been invented in different domains. While there are 
different standardization processes regarding constitutional values, there is yet 
no specific standard in dealing with AI and democracy. To date, nothing specifies 
a general obligation to include the principle of democracy into the design of AI.” 


D. Legal Reasons and Lessons for the Democratization of Al 


This section sketches the main legal reasons for democratizing AI as well as some 
learnings from the relationship of law and democracy. Democracy as a principle 
is enshrined in the constitutions of many states, be it implicitly or explicitly; it is 
also a basic value for international organizations such as the Council of Europe.” 
Such a constitutional principle demands its realization in the public sphere. Apart 
from this very general democratic requirement, there are more specific lessons 
that can be drawn from the way that law functions. Three insights will be dis- 
cussed in greater detail below. 


30 Nemitz (2018). 
31 Current value-sensitive design standards can be found with the respective ISO projects and in 
IEEE’s P7000 series. 


32 See for example the preamble of the statue of the Council of Europe from 5 May 1949, ETS No. 
001. 
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1. Justification 

As mentioned above, human rights add another layer to the limitation of technolo- 
gy. They set absolute limits on the behavior of public authorities and force them to 
realize human rights. Human rights are also tied to democratic decision-making. 
Whenever a measure touches upon human rights, it can only be lawful when there 
is a democratic justification underpinning it. The Covenant on Civil and Political 
Rights uses the phrase “rights shall not be subject to any restrictions except those 
which are provided by law.” Meanwhile, the European Convention on Human 
Rights uses the phrase “in accordance with the law.” This means that restrictions 
of human rights must be provided by law.” In order to qualify as a justification, 
the impact must be described by law in a manner that is understandable for the in- 
dividual. The law here is a proxy for a democratic ex ante decision-making. Any im- 
pact upon human rights must be preceded by a democratic decision allowing for 
the precise impact and providing for safeguards for excessive and arbitrary uses. 
Another example is the Charter of Fundamental Rights of the European Union 
which provides that “[alny limitation on the exercise of the rights and freedoms 
recognized by this Charter must be provided for by law.”* This provision makes 
the need for democratic justification explicit. In the absence of such a justification, 
a measure is necessarily unlawful. The law is a vehicle to enforce human rights. It 
is also a medium for democratic decisions. This strong link between human rights 
and democracy mediated by the law also affects the relationship between AI and 
democracy. Whenever AI systems have an impact on human rights, their use is to 
be justified. 

This necessity for democratic justification does only depend on the fact that 
the human capacity to make decisions affected by an AI system. Democratic justi- 
fication is not only triggered by specific human rights. The need for justification of 
Al systems certainly applies to so-called automated decision systems (ADMs) that 
are often in the focus of academic attention. This is only one among many ways in 
which human rights can be at issue. When AI is used as a watchdog for IT securi- 
ty or for maintenance of critical infrastructure, it is crucial for realizing human 
rights. While the right to privacy and self-determination might be the most obvi- 
ous examples of such impacts, other subtler influences also need to be considered. 
For example, ADMs are frequently regulated, but this regulation never applies to 
recommendation systems. Yet, these recommendation systems can have substan- 
tial impacts on human rights. Highlighting the interdependence between artifi- 


33 See for example the explanation by Greer (1997: 9). 

34 Charter of Fundamental Rights of the European Union, OJ C 326, 26.10.2012, p. 391-407. The re- 
lationship between legal democratic justification and human rights is not as universal in every 
human rights instrument. The universal human rights covenants, for example, require legal 
democratic justification only in certain cases. 
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cial intelligence and human rights, there is a rather clear criterion for the need of 
democratic determination. This is not the “power” of a machine to decide, but the 
impact on human rights. The interrelation between human rights and democracy 
can limit the public use of AI systems. If there is such a relationship, democratic 
justification is necessary—irrespective even of how human rights are affected. 


2. Supremacy 

Tied to this necessity for democratic justification issues impacting human rights 
is the idea of the supremacy of legitimate democratic decisions. This concept has 
found its expression in the idea that the norms made by the organization with 
the highest democratic legitimacy take precedence over other norms. Therefore, 
many jurisdictions which regard parliament as the highest democratic authority 
rely on the “sovereignty of parliament” and regard parliamentary laws as taking 
precedence over all other legislative acts. Other jurisdictions describe a norma- 
tive hierarchy in which the constitution is at the top and acts of parliament in the 
second place. While constitutional law derives its legitimacy from the pouvoir con- 
stitutant, statutory law relies on the legitimacy of parliament and yet other norms 
stem from actors with less legitimacy. Higher norms take precedence over lower 
norms, in cases of conflict, lower norms are either rendered invalid or inappli- 
cable. The hierarchy of legal norms is generally grounded in different levels of 
democratic legitimacy.” In cases in which technology has normative force, this 
general idea would require that the law as a proxy of democratic decision takes 
precedence over functional requirements of technology and must actually guide 
democratic decisions.’‘ 


3. Democratic Rebalancing 

From a legal point of view, the notion of democracy is open. While there are many 
ways to understand and construct what democracy ought to mean, constitution- 
al law is generally open towards the multiple understandings and theories of de- 
mocracy. This openness allows the law to adapt to different contexts and different 
situations, especially when changes and reforms are at issue. Such reforms can 
happen on different levels, but they always change prior democratic processes and 
sometimes even the notion of democracy itself. One pattern that can be discerned 
from the way in which courts deal with these issues could be described as the 
mode of rebalancing. Courts remain flexible and open towards changing existing 


35 From a legal positivist standpoint, it would also be possible to arrive at the same conclusion ar- 
guing with validity. One would then have to argue that the basic reason for validity is democracy. 

36 See for example Schulz/Dankert (2016) It is important to note, however, that such a hierarchy 
must be based on democratic legitimacy and not on a formal distinction of primary and sec 
ondary rules. 
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processes, but they require active steps that would rebalance the situation from a 
democratic standpoint. This rebalancing can mean that there are measures that 
effectively democratize the new institutional arrangements. Two examples from 
other contexts can illustrate this. In the process of European integration, there 
were many treaty revisions creating new competences or transferring competenc- 
es from the national to the European level. The German Federal Constitutional 
Court had to deal with creation and transfer of competences on several occasions. 
In its famous Lisbon judgment, the court allowed for a transferal of competenc- 
es, but it also required institutional arrangements in the German legal order, en- 
abling the legislature to effectively play a role in European politics. So, while it 
agreed to supranational power transferals, it only did so on the condition that the 
national legislature could influence politics at the higher level.” In another case, 
the Constitutional Court of Baden-Wiirttemberg had to deal with a transferal of 
powers from the collegiate of professors to the president of a university. The court 
allowed for this transferal of power, but only on the condition that the president 
become accountable to the collegiate of professors, which in practice meant that 
a democratic election process had to be created.** These cases show that changes 
and reforms with an impact on democratic processes are—from a legal stand- 
point—not to be evaluated in a binary fashion of “yes” or “no.” Changes some- 
times require democratic rebalancing. If there are disputes on how to rebalance 
those changes democratically, those disputes can ultimately be resolved in legal 
proceedings. These questions of rebalancing play an important role when actions 
and decisions are delegated to AI systems on a greater scale. Instead of arguing 
that this would be undemocratic, the question is rather whether this delegation to 
AI systems can be rebalanced. This flexible view present in different democracy 
cases also has the potential to shift the relationship between AI and democracy. 
Instead of asking whether AI should be democratized, the question is how it can 
be democratized and whether the respective measures are enough. 


Il. How to Democratize Al 


If there is a need to democratize AI, how can it be put into practice? An instru- 
mental approach to that question would first look at instances in which there are 
democratic choices and secondly at ways in which these decisions can be made. As 


37 BVerfG, Judgment of the Second Senate of 30 June 2009—2 BvE 2/08—paras. (1-421), http:// 
www.bverfg.de/e/es20090630_2bveooo208en.html para 273ff. 

38 Landesverfassungsgericht Baden-Württemberg, judgment 14.11.2016, 1 VB 16/15, obtaible at 
https://verfgh.baden-wuerttemberg.de/fileadmin/redaktion/m-verfgh/dateien/161114_1VB16- 
15_Urteil.pdf p. 43ff. 
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in every other democratic decision, there are different tools ranging from the or- 
dinary processes of parliamentary deliberations and decisions to more direct ver- 
sions of democratic participation. Each method might have specific advantages in 
a certain setting. Such a democratic toolbox could contain the following elements 
among others: 


« ordinary parliamentary processes to debate and regulate artificial intelligence 
e use of specialized parliamentary committees to determine certain issues 
e empowerment of experts to make certain decisions according to preconfig- 
ured principles 
e direct involvement of citizens regarding certain questions through 
° participatory methods 
°  sortition: involving groups of randomly selected citizens in order to fulfil 
an office or make certain decisions 
° random sample voting: in order to vote on specific questions, a representa- 
tive sample of the population is selected 


For the sake of understanding the range of choices to be made about technologies 
and specific technical artefacts, it is helpful to distinguish between different lay- 
ers analytically, despite the fact that the interrelations between the different lay- 
ers are obvious. Focusing on specific choices regarding technical artefacts, there 
are choices that are rather technical and others that are rather social. Therefore, 
a distinction is to be made between a social and a technical layer. Furthermore, 
some decisions are not made with a view to a specific artefact but rather regarding 
a technology. These choices are situated in a layer of governance. On every layer, 
there are specific questions to be outlined. 


A. Technological Layer 


1. Design Choices 

An important step in the democratic determination of technology is understand- 
ing the choices that are made in the course of inventing or applying a technology. 
Many design choices are made in the development. Some design choices are made 
intentionally, some have important consequences. From a democratic perspective, 
one must understand and highlight specific choices. These choices relate to archi- 
tectures, applications and all other features of the technologies used. Whenev- 
er there is an alternative, there is a choice. Understanding choices also requires 
a democratic mindset that is open to several possibilities without automatically 
preferring certain outcomes. Computer scientists especially, who are trained to 
achieve specific goals such as efficiency, regularly do not see behind the choices 
that maximize their preferred value. 
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In order to appreciate choice in the case of machine learning, questions of 
optimization are very interesting.” Machine learning systems are optimized to 
attain certain goals, they receive feedback and adjust their model accordingly. In 
many cases, the goals towards which a model is optimized are not set in stone 
but rather contingent. An algorithm that distributes children to certain schools 
within an area can be optimized according to different goal functions: One could 
be the shortest way to school. Another would be the safest way to school. Yet, one 
could also define other goals such as a good mix of students in school from an 
ethnical or economic perspective. Such choices often result in trade-offs. They 
require an active choice. One trade-off that has become better known as of late is 
the choice between using data and being able to understand discrimination. Ma- 
chine learning models are often trained on data that contains implicit biases—at 
the same time, training data may not contain explicit references to age, gender or 
other criteria. Thus, the decisive information is not present and it becomes impos- 
sible to understand whether there is bias in the data and consequently also in the 
algorithm and whether remedies are possible. Yet, including more data, e.g. age 
or gender, impacts the right to privacy and data protection. Especially in possible 
cases of discrimination, it would often be necessary to use special categories of 
personal data, such as data revealing racial or ethnic origins, that is heavily pro- 
tected under many data protection regimes.*° Therefore, it is necessary to weigh 
privacy and data protection against fairness in this regard. Another trade-off can 
happen when it comes to weighing transparency and accuracy. It is possible that 
some algorithms have higher scores than comparable alternatives but are based on 
models so complex that they are not intelligible for humans. There is an increas- 
ing awareness in the computer science community that choices are not only made 
in the process of using existing technologies but also in the process of research 
and development. In the same way that privacy enhancing technologies were in- 
vented, new communities have sprung up doing research to improve Al in specific 
directions. One example is the ACM Conference on Fairness, Accountability, and 
Transparency (ACM FAT), that looks specifically at new research on fairness, ac- 
countability and transparency in socio-technical systems. Similar conferences or 
tracks on AI panels show how research and development can also be specifically 
directed towards certain aims. Again, there is an element of choice even when it 
comes to creating or improving technologies. In this case, these choices can be 
exercised by researchers, but also influenced by funding agencies. An element of 
choice is often present at different stages. 


39 Haferkamp (2017). 
40 See for example Art. 9 GDPR. 
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2. The Principle of Designability 

Scholars and institutions have called for the inclusion of democracy by design in 
the context of AL.“ In line with the idea of value-sensitive design, democratic val- 
ues should be included in the design process. Not only are design choices to be 
made in a democratic manner, the very way in which the application operates is to 
be democratic. Yet, this general idea encounters several difficulties. One problem 
is that there are varying concepts of democracy and they can play out quite differ- 
ently in the design of an application.” One way to structure the different forms of 
democratic legitimacy is to divide them into input, output and process legitimacy. 
Technical requirements can be quite sophisticated. Depending on the context in 
which the AI application is used, democracy can involve very different actors as 
well: in the smart city context, democratic decisions will often require decision 
making or participation by the municipal population. In national settings, it will 
be more about involving parliament in decisions. For these reasons, the assertion 
of democracy by design means a lot of uncertainty for developers. What would 
be needed from a technical perspective is a principle that developers can grasp 
and one that supports democratic values in design processes without prejudging 
certain understandings of democracy. 

My suggestion to address this challenge would be to formulate a design prin- 
ciple of designability. The principle of designability is aimed at translating gen- 
eral democratic values into design in a general and workable manner. It ought to 
have at least two tiers that need to be addressed by developers: The first tier is the 
changeability of the system. The second tier is its intelligibility. Different ideas of 
democracy rely on the idea that they are open and flexible to different forms of 
change: changes in government, changes in opinion after an informed discourse 
and so on. This is particularly the case if there is uncertainty about how a decision 
plays out in practice. In such a situation, changeability is a requirement for dem- 
ocratic participation. Yet, such changeability has to be enhanced by design. This 
can be done by choosing a specific architecture or using specific methods. Con- 
sidering that machine learning entails the possibility to adapt, it is changeable by 
definition. Another tier for designability is the intelligibility of the system. Intel- 
ligibility is not used in its general sense in computer science, that is the possibility 
to understand the logic behind a given system’s actions. Intelligibility must be 
constructed democratically. A general target here could be that a system is intel- 
ligible for all people affected by the actions of the system. While not everybody 
will in effect decide upon whether and how to employ the respective AI system, 
the ideal would be that everybody should have the chance to. This standard of in- 


41 See for example Nemitz (2018); Die Bundesregierung (2018: 33, 44); High Level Expert Group on 
Artificial Intelligence (19). 
42 Bozdag/van den Hoven (2015). 
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telligibility can be rather narrow in the case of systems that are targeted only at a 
specific group of people. In contrast, generally applicable AI systems should meet 
general standards of intelligibility. Therefore, the tier of democratic intelligibility 
fits in with current discourses on transparency. Yet, in the context of designability, 
intelligibility is not limited to specific actions or decisions made by the system. 
The people affected by the system have to understand it and the choices underly- 
ing it. They have to know whether and how the system can be changed. Like any 
design principle, designability will hardly ever be achieved fully. But it can point 
developers in the right direction. While intelligibility points to the possibility 
of democratic deliberations, the tier of changeability indicates the possibility of 
change and opens up potentials to effectively govern the technical artifact. 


B. Social Layer 


Alis not only designed on the technical level, many social constructions surround- 
ing Al systems play a crucial role.” These social constructions are not inevitable, 
they are the fabric of choices and assumptions that are shaping technology and 
society at the same time. The law is a mechanism that can make socio-technical 
choices subject to democratic determination. 


1. Understanding Impacts 

It is important to appreciate the social impact of technology, but also to under- 
stand that the recognition of such impacts are social constructs themselves. Re- 
cently, different methods to assess the impacts of AI have been proposed.“ Im- 
pact assessment is a prerequisite for uncovering choices on the technological level. 
Sometimes, the respective choices only become apparent and understandable 
when the social impacts are known. The discussion about fairness in AI took off 
when several researchers criticized discriminatory effects of algorithmic systems. 
The same is true for transparency. To learn about the consequences of technolo- 
gies before harm and damage occurs is far from easy. As the history of technology 
shows, the knowledge about the consequences of technologies often comes too 
late. The discovery of radiation is a telling and sad example, since many of the 
scholars discovering this technology did not know about its dangerous effects and 
later died from cancer. It took some time to understand the effects. In many other 
instances, the causal relationship between technology and impact was not as ap- 
parent or more contested. In these instances, the law has profound effects on the 
social construction of technology. 


43 Stamper (1988). 
44 Reisman/Schultz/Crawford/Whithattaker (2018); ECP (2018). 
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Firstly, human rights law can provide for a consensus that a certain consider- 
ation is worthy of protection. In order to know what constitutes an impact, one 
has to construct a value that is to be protected. The law can create a consensus of 
what that is. The right to privacy is a good example ofa right that has been invent- 
ed through deduction in an evolutionary manner from other legal institutions.” 
Once there is an agreement on what is to be protected as human right, a special 
protection is in place. As has already been shown, this protection entails the need 
for democratic justification of decisions affecting human rights. Another import- 
ant feature of the law is its ability to recognize and balance impacts in a holistic 
manner. Impacts are not negative by definition. They can equally be beneficial. 
While it is important to be critical towards new developments and to understand 
new dangers and disadvantages, it is as important to appreciate the benefits and 
potential opportunities. In order to assess the impacts of technologies, it is cru- 
cial to have all of the future possibilities in mind. This is also true from a human 
rights perspective. As shown above, technologies also have the potential to further 
human rights. Therefore, the consequences have to be weighed against each other. 
In order to assess such situations in legal proceedings, several jurisdictions have 
developed a proportionality test.* It is a practical way to assess a measure holisti- 
cally and to structure the argument in a way that allows for many considerations 
and to weigh them against each other. It also arrives at practical conclusions that 
are communicated to those affected by the decisions. The principle of proportion- 
ality actually allows for a socio-technical evaluation on different levels. 


2. Designing Al through Social Construction 

Yet, there is an even wider sense in which the impacts of Al are socially construct- 
ed. This applies to a large part of the influence of AI systems. Especially in the case 
of data analytics, there can be different goals and aims: to discover certain cor- 
relations, to discover probabilities of certain actions or to actually show probabil- 
ities of how certain alternative actions might play out.” While it is true that those 
systems can have profound normative effects, such effects often stem from the 
social construction of the system instead of being falsely pinpointed as inherent 
in the technology. Whereas big data analytics tools compute certain probabilities, 
for example, the meanings of those probabilities and the role they should play is 
actively constructed.* One illustrative example is the misuse of scores for cred- 


45 See one early argument in Warren/Brandeis (1890). 
46 Klatt/Meister (2012). 


47 Onthis basis a distinction is made between descreptice, prescrpitive and descriptive analytics 
by Hoffmann-Riem (2017). 
48 Seeforexample Schlaudt (2018). 
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itworthiness as a reliability score for employees.” It is obvious that a system that 
is designed to compute the probability of a person repaying debts is not made to 
assess the respective persons reliability when it comes to the job. Yet, the choice 
to use the system in another context is by no means a choice that has anything 
to do with the design of the system. It is rather a social choice for a transfer to a 
different social context. 

The same holds true for the use of certain probabilities. In many instances, the 
law shows how probabilities have completely different meanings in different con- 
texts. In police and security law, there are also different probability requirements 
that are formulated from a social perspective. Measures that have low impacts 
on human rights have to meet a lower probability threshold, while measures with 
higher potential impacts have to meet higher probability standards. It is an active 
choice, and a democratic decision, to link a specific competence of the authorities 
to acertain probability. 

There are numerous ways in which to construct the meaning of outputs of AI 
systems. The law not only makes this meaning explicit; it opens up the social con- 
struction of technology for democratic deliberation and democratic decision-mak- 
ing. The outputs of AI systems can be rendered illegal and irrelevant. They can 
be made subject to human oversight and human decision-making. Furthermore, 
they can be bestowed with the force of the law. In German law the assessment 
of civil servants, decisions must not be based on fully automated assessments of 
specific personality features." The above-mentioned Art. 22 GDPR provides for a 
right of human oversight and makes fully automated decisions subject to human 
decisions. Yet, there are provisions clarifying that fully automated decisions do 
have the force of law. Take for example $ 35a of the Federal Code of Administrative 
Procedure. The provision states: “An administrative act may be adopted in full by 
automatic systems, provided that it is authorised by a legal act and that there is 
no discretion or margin for assessment.” This provision clarifies that there can be 
completely automated administrative acts, i.e. decisions with legal force for spe- 
cific individuals or groups. This basically means that those systems can render de- 
cisions that have the force of law and can also be enforced. Two examples for such 
decisions are intelligent traffic systems that can automatically set speed limits or 
impose overtaking bans when there are dangers for the drivers due to wheather or 
traffic. Another example is fully automated speeding tickets issued from detec- 
tion systems that automatically send the respective notices. 


49 O'Neil (2016: 147-149). 


50 See §114 section 4 of the German Federal Civil Servants Law. 
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3. Alas Customary Law 

AI systems can have real world impacts which depend to a large extent on a social 
construction that attributes these consequences to the system. This leads to the 
question of what the requirements of such acceptance should be. This question is 
currently addressed by the field of computational social choice.” The hidden mor- 
al choices in the process of designing AI is one of the main motivations to engage 
with the interlaces of social choice and computer science. So, the proponents of 
computational social choice try to find criteria to design AI systems in a legitimate 
way. One feature that is striking with machine learning is that it is actually based 
on data that is often produced by those to whom the system applies. Research 
projects have, for example, used inquiries and simulations in order to obtain user 
data on how automated cars should react in specific situations.” Yet, a democratic 
view on this ethical design focus reveals certain issues: The first problem is that 
different assumptions can lead to quite varied results, which might all have a 
claim to be ethical. Different ethical theories can even produce opposite results. 
Take for example utilitarianism and principled ethics. While certain actions det- 
rimental to one person but beneficial for the majority could be regarded as ethical 
from a utilitarian perspective, they would be regarded as unethical from a prin- 
cipled point of view. In the end, it might be necessary to choose among many al- 
ternatives. To state that there is only one right and moral solution to be preferred 
over all other solutions is to discriminate against all other possible solutions. It 
neglects various approaches and different solutions to a single question. In such 
a setting, there is no room left for choice. Another question is whether artificial 
agents can genuinely make moral decisions or whether they are just simulating 
them. From a moral point of view, the question of actual judgement is paramount. 
This problem is tied to the question whether machines can actually think, which 
has attracted contentious reflection from Turing to Searle.” 

The basic argument of this section is that computational choice theorists should 
think in legal instead of moral terms. Building upon Kant, one could attribute ac- 
tions with external effects to the law, while questions that remain internal are 
in the realm of ethics. AI systems often have profound normative effects. While 
most ethical considerations focus on output legitimacy, one could merge compu- 
tation and law in a way that democratic input legitimacy is achieved through legal 
means. Machine learning applications are generally trained with data that rep- 
resents the behavior of certain actors. While there is no general formalized rule 
about what the significance of such practice is, I would like to make the argument 


51 Brandt/Conitzer/Endriss/Lang/Procaccia (2016) A overview of the literature regarding Al is giv- 
en by Prasad (2019). 

52 Awad/Dsouza/Kim/Schulz/Henrich/Shariff/Bonnefon/Rahwan (2018). 

53 Turing (1950); Searle (1980). 


Al, Democracy and the Law 


that machine learning could—under certain conditions—be regarded as custom- 
ary law. This would highlight computational and social choices that allow for a 
democratic expression through an Al system. Building upon an analogy from cer- 
tain law-creating practices, it could be possible to formulate requirements for AI 
as a medium for democratic decisions. 

Customary law used to play a very important role for the governance of cer- 
tain communities that regarded specific practices as binding. Spurred by the 
increasing complexity of modern societies and the possibilities of new printing 
technologies, customary law lost much of its importance. It mainly relied on un- 
written practices of smaller communities that formed over time. While courts in 
the common law countries continued to rely on once formed principles and turned 
them into arguments the judiciary could build on, one legal system in which cus- 
tomary law has retained its importance is international law. In international law, 
there is still a manageable number of participants whose practice can be quali- 
fied as custom. Several trends of digitization assist a new knowledge dimension 
that might lead to a revival of customary law in different areas. First, datafication 
opens new avenues to store and understand the behavior of certain actors. Big 
data represents the idea that huge amounts of data can be stored and analyzed. 
Secondly, trends like the internet of things allow for the collection of data in a con- 
stant, automated and ubiquitous manner. The internet of things signifies a trend 
of networked devices in different human environments. AI technologies can help 
to analyze and understand the data in a way that makes the practice comprehen- 
sible and understandable. Together, those technologies make actual practice of 
people visible. 

However, the question remains as to whether this custom is meant to be gen- 
eralized in human exchange. Scholars of computational social choice have thought 
about this issue and come up with criteria that were to be considered in the pro- 
cess of building an AI that represents practice. Baum, for example has developed 
with three general criteria: 


1. Standing: Who or what is included in the group to have its values factored into 
the AI? 

2. Measurement: What procedure is used to obtain values from each member of 
the selected group? 

3. Aggregation: How are the values of individual group members combined to 
form the aggregated group values?* 


The requirements of customary law are in some sense complementary, in some 
sense different from the questions above. The formal criteria for the formation of 
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customary law are a practice (consuetudo) and the belief that this practice is to be 
regarded as law (opinio iuris sive necessitatis). The practice must be consistent and 
general, even though this does not mean that the practice is uniform and uni- 
versal.” The most important question regarding general practice in the context 
of customary law is whether there is sufficient representation. This is due to the 
fact that some actors remain tacit and do not engage in the practice. The second 
criterion is the so-called opinio iuris. That is the belief that the respective practice is 
based upon a legal obligation to act in that way. This criterion actually legitimizes 
the normative force of the practice. In order to fulfil the criterion of opinio iuris, 
data subjects must produce the data in the knowledge with the purpose of influ- 
encing a system that acts upon that data. This criterion makes the legitimacy of an 
AI system subject to a sovereign decision of users. The system simply learns what 
the practice of human beings is. It learns what the data subjects want the practice 
to be. In this setting, informational self-determination is not only the power of 
personal data; it is a conscious exercise of power through one’s data. The data sub- 
ject is not a resource from which personal data are extracted. In this setting, the 
production of data becomes a democratic act like voting. 


C. Governance Layer 


In order to analyze the impact of AI on democracy, it is not enough to look exclu- 
sively at specific systems. It requires an analysis from the macro level focusing on 
technologies or even AI as a whole. This is here denoted as the governance layer. 


1. Framing 

The democratic governance of AI is influenced by the way in which AI is framed. 
Alis regularly put in specific contexts or seen a certain way. Frequently, scholars 
talk about the ethics of AL,” another current is to talk about Aland human rights. 
While scholars discuss and analyze within one frame, there is relatively little dis- 
cussion about the choice between frames. Yet, the frames do have significant ef- 
fects. Take for example the choice between an ethical and a political frame.” The 
frames lead to completely different ways of thinking about technology. Compare 
stem cell engineering and the creation of a 5G network infrastructure. Stem cell 
engineering is predominantly construed as an ethical issue whereas the latter 
is commonly perceived as a political issue. Of course, there are many issues we 
would conceive of as being political in the context of stem cell research and there 
can be many ethical questions in building a 5G infrastructure. Constructivist 
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scholars have highlighted that frames and theories influence the object of scien- 
tific inquiry. Therefore, it is an active choice to put AI in certain context and to 
inquire into the ethics or politics of AI or to look to the relationship of AI and hu- 
man rights. This choice necessarily contains certain preferences that are inherent 
or follow from the frame that was adapted. Every frame also provokes some blind 
spots. Some aspects become invisible. 
One attempt to generally describe the impact of AI on society is the concept of 
“algocracy.” This term contrasts other forms of government such as democracy or 
monarchy with a system in which power is (increasingly) exercised by automated 
systems.” The term algocracy is mostly used in a critical manner.” It highlights 
that algorithms are becoming more and more important when it comes to issues 
of governance. Instead of adding to the growing corpus of literature on this issue, 
I would like to highlight the constructivist nature of algocracy. This leads to the 
question of what is highlighted by this term and what is left out of the picture. 
Building on the basic insights from actor network theory (ANT), I argue that the 
frame of algocracy tends to blur and hide human agency. Algocracy highlights 
machine power but overlooks how humans impact the perceived automated ac- 
tions. One of the basic arguments of ANT is to ignore the distinction between 
subjects and objects and to appreciate technology as part of the social in a net- 
work with human actors using it.® This analysis allowed the proponents of ANT 
to uncover the agency of technical artefacts. My basic argument is that this theory 
might today be used upside down in order to uncover human agency instead of 
machine agency. The theory of algocracy represents a critical part of the AI dis- 
course that frames AI specifically as automated decision systems and looks at their 
increased power. With the focus on increasing ability and power of those systems, 
it is sometimes forgotten to reflect on how these systems are used and interwoven 
with human agency. As outlined above, there are many ways in which the social 
surrounding determines the design of AI applications. In many cases, the law is 
part of constructive efforts to bestow AI with normative force. A frame that is 
complementary to algocracy would not exclusively look at the fact that more and 
more decisions are delegated but at how they are delegated and who controls and 
influences the automated systems. As many proponents of ANT have argued, the 
focus would not be ona single class of actors but rather on their interrelation. 
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2. Organizational aspects 
Another way to impact the development and deployment of artificial intelligence 
is through organizational measures. Many of the recent Al strategies contain such 
measures. On the one hand, organizational changes are aimed at enhancing tech- 
nological progress in the field of AI. New institutions are founded, either to en- 
gage directly in research and development, to fund such activities or to enhance 
the network of already existing organizations. The United Arab Emirates made 
headlines with a minister for artificial intelligence“ and the German government 
recently founded an agency for “innovation leaps” tasked with funding research 
and development for ground-breaking innovations and increasing implementa- 
tion. On the other hand, newly founded organizations also exercise oversight over 
AI systems. In fact, there are indeed many organizations endowed with this task 
already. Organizations like the US Federal Drug Administration or its counter- 
parts in Europe and elsewhere have engaged in the certification of AI systems that 
are considered to be medical products. There are also calls for more oversight in- 
stitutions.” Following examples in Canada, some states have founded AI observa- 
tories that aim to find out about the social consequences of AI. The future of work 
is one of the issues often addressed in this context.® 

Organizational change is not always expressed merely in new organizations. 
Sometimes, organizations change from within by adapting to new tasks. One im- 
portant development in this regard is the question whether a new job profile is 
needed across organizations. Data scientists are one profile that is currently on 
the rise. Yet, some think that a completely new profile of algorithmists might be 
needed.‘ The idea behind this is to have people with specific technical skills so 
that an organization maintains agency when it must deal with AI systems. The 
interesting aspect of this idea is that expertise would also be available to orga- 
nizations that have previously not been associated with technological expertise. 
The job profile of an algorithmist has the potential to democratize agency when it 
comes to questions of algorithms. Knowledge about AI systems would be gener- 
ally available. A question separate from this specific profile would be the interdis- 
ciplinary mix of teams working on certain Al issues. If AI is used in specific con- 
texts, there might be a minimum requirement of roles and perspectives that need 
to be present. Therefore, organizations developing, using or assessing AI systems 
should think about what the right mix of these teams would be. While computer 
scientists are a necessary component of such teams, they are never enough. All in 
all, organizational challenges and changes are a very good example of how algo- 
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rithms impact their social surroundings and how changes in the socio-technical 
context of AI systems can effectively contribute to the respective governance. 


Ill. Conclusions 


The 1947 constitution of Bremen, an entity of the German federal state, contains a 
very interesting provision about the relationship between man and machine. The 
constitution states in Art. 12 section 1: “The human being ranks higher than ma- 
chines and technology.” This provision addresses experiences from the process of 
industrialization, during which machines, technologies and the new possibilities 
of production gained importance. It is interesting that the founders of the con- 
stitution felt the need to remind the people and those in power of the fact that 
human beings should rank higher. During industrialization, this did not address 
the increasing capabilities of machines to act so intelligently that they may even 
be considered as persons. It was rather the fact that, as capacities of production, 
so much importance was conferred upon them. So, the basic idea was to argue 
for a human-centered view despite the huge social and economic importance of 
technical artefacts. This basic idea can also be translated to the process of digi- 
tization, in which machines engage in solving problems that require a degree of 
intelligence previously considered exclusively reserved for humans. One aspect of 
this normative centricity of human beings is their exclusive status as the bearers 
of human rights. Equally important is the aspect of effective self-determination 
of people in the face of technologies’ increasing possibilities. To rank higher does 
not only mean that humans must not be harmed by new technological possibili- 
ties. It means that people need to be in the driver’s seat. It can be understood as a 
call for effective self-determination on different levels. 

If AI continues to fulfil the high expectations and has continued impacts on 
societal development, it will be even more important for an all-encompassing 
value-sensitive development. From the perspective of the constitution of Bremen, 
one necessary component would be to think about the democratization of AI. In 
order to do this, it will be crucially important to understand AI as a set of gen- 
eral-purpose technologies that can be used in very different circumstances and 
very different ways to achieve multiple tasks. While it is important to understand 
where AI currently threatens democracy, it is as crucial to appreciate its opportu- 
nities. To understand the openness of the use and potential of technology allows 
us to choose whether to develop the technology further and which path to take. 
When it comes to the democratization of AI, some general truths about democra- 
cy apply: Democracy is a process, not an achievable result. It can be lost very easily, 
and everyone must work for it continuously along the way. Once we stop striving 
for it, it is gone. From this perspective, AI is just another challenge that has the 
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potential to bring society closer to the ideal lying behind Art. 12 section of the Bre- 
men Constitution, as well as many other democratic provisions: to meaningfully 
put people in the normative center of all public power. 
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Rethinking the Knowledge Problem 
in an Era of Corporate Gigantism' 


Frank Pasquale 


A preeminent theorist of laissez-faire, Friedrich von Hayek called the “knowledge 
problem” an insuperable barrier to central planning. Knowledge about the price of 
supplies and labor, and consumers’ ability and willingness to pay, is so scattered 
and protean that even the most knowledgeable authorities cannot access all of it. 
No person knows everything about how goods and services in an economy should 
be priced. No central decision maker can grasp the idiosyncratic preferences, val- 
ues, and purchasing power of millions of individuals. That kind of knowledge, 
Hayek said, is distributed. 

However, in an era of artificial intelligence and mass surveillance, the allure 
of central planning has reemerged—this time, in the form of massive firms. Hav- 
ing logged and analyzed billions of transactions, Amazon knows intimate details 
about all its customers and suppliers. It can carefully calibrate screen displays 
to herd buyers toward certain products or shopping practices, or to copy sellers 
with its own, cheaper, in-house offerings. Mark Zuckerberg aspires to omni- 
science of consumer desires, profiling nearly everyone in Facebook, Instagram, 
and Whatsapp, and then leveraging that data trove to track users across the web 
and into the real world (via mobile usage and device fingerprinting). Indeed, you 
don’t have to use any of those apps to end up in Facebook/Instagram/Whatsapp 
files—profiles can be assigned to you. Google’s “database of intentions” is legend- 
ary, and antitrust authorities around the world have looked with increasing alarm 
at its ability to squeeze out rivals from search results once it gains an interest in 
their lines of business. Google knows not merely what consumers are searching 
for, but also what other businesses are searching, buying, emailing, planning—a 
truly unparalleled match of data processing capacity to raw communication flows. 

Nor is this logic limited to the online context. Concentration is paying div- 
idends for the largest banks (widely assumed to be too big to fail), and major 
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health insurers (now squeezing and expanding the medical supply chain like an 
accordion). Like the digital giants, these finance and insurance firms not only act 
as middleman, taking a cut of transactions, but also aspire to capitalize on the 
knowledge they’ve gained from monitoring customers and providers in order to 
supplant them and directly provide services and investment. If it succeeds, the 
CVS-Aetna merger betokens intense corporate consolidations that will see more 
vertical integration of insurers, providers, and a baroque series of middlemen 
(from pharmaceutical benefit managers to group purchasing organizations) into 
gargantuan health providers. A CVS doctor may eventually refer a patient toa CVS 
hospital for a CVS surgery, followed up by home health care workers employed by 
CVS who bring CVS pharmaceuticals—all covered by a CVS/Aetna insurance plan, 
which might penalize the patient for using any providers outside the CVS net- 
work. While such a panoptic firm may sound dystopian, it is a logical outgrowth of 
health services researchers’ enthusiasm for “integrated delivery systems,” which 
are supposed to provide “care coordination” and “wraparound services” more effi- 
ciently than America’s current, fragmented health care system. 

The rise of powerful intermediaries like search engines and insurers may seem 
like the next logical step in the development of capitalism. But a growing chorus 
of critics questions the size and scope of leading firms in these fields. The Institute 
for Local Self-Reliance highlights Amazon’s manipulation of both law and con- 
tracts to accumulate unfair advantages. International antitrust authorities have 
taken Google down a peg, questioning the company’s aggressive use of its search 
engine and Android operating system to promote its own services (and demote 
rivals). They also question why Google and Facebook have been acquiring at least 
two companies a month, for years. Consumer advocates complain about manipu- 
lative advertising. Finance scholars lambaste megabanks for taking advantage of 
the implicit subsidies that too big to fail status confers. 

Can these diverse strands of protest and critique coalesce into something 
more durable and consistent? This essay explores possible forms to channel social 
and economic discontent over the next few decades. I start by giving an account- 
ing of where we are: a hierarchical, centralized regime, where corporate power is 
immense, and where large national apparatuses of regulation seem to be the only 
entities capable of reining it in. Against this economic reality, I can discern two 
vital lines of politico-economic critique at present. 

Populist localizers want a new era of antitrust enforcement to rein in giant 
firms. These Jeffersonian critics of big tech firms, megabanks, and big health care 
combinations are decentralizers. They believe that power is and ought to be dis- 
tributed in a just society. They promote strong local authorities, who are located 
closer to their own citizens. 

Others have promoted gigantism as inevitable or desirable, and argue that we 
simply need better rules to cabin abuses of corporate power. Today’s Hamiltoni- 
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ans argue that massive stores of data are critical to the future of artificial intelli- 
gence—and thus, to productive dynamism of the economy. They focus on better 
regulating, rather than breaking up, leading firms. 

Jeffersonians and Hamiltonians have very different long-term views on what 
an optimal economy looks like. In the long-run, their visions are probably irrecon- 
cilable. However, in the short run, both sets of reformers offer important lessons 
for policymakers grappling with the power of massive tech, finance, and health 
care firms. This essay explores those lessons, specifying when a Jeffersonian ap- 
proach is most appropriate, and when Hamilton’s heirs have the better approach. 


The Jeffersonian/Hamiltonian Divide 


The tech policy landscape is often bleak. Corporate-funded think tanks strive to 
keep reform options in a relatively narrow window of tweaks and minor changes 
to existing law. The curse of over-specialization in the academy also keeps many 
law and policy professors on a short leash. Nevertheless, there are pockets of vital- 
ity in the field. Two camps that have arisen include a decentralizing camp, which 
I'd call Jeffersonian, and a more centralizing, Hamiltonian tendency that is com- 
fortable with industrial “bigness.” 

The Jeffersonian school has coalesced around the problem of lax antitrust en- 
forcement in the United States, and competition promotion more generally. The 
Open Markets Institute, kicked out of the New American Foundation for being too 
hostile to Google, has led the charge. Leaders at OMI, like Matt Stoller and Barry 
Lynn, argue that the Federal Trade Commission (FTC) should break up Facebook, 
establishing Instagram and WhatsApp as competing social networks. Lina Khan, 
also at OMI, has written an exhaustive critique of Amazon’s gigantism that is 
already one of the Yale Jaw Journal’s most downloaded articles. The emphasis on 
subsidiarity in Catholic Social Thought is also a font of decentralist theory, often 
invoked by conservatives to protect the autonomy of local authorities and civil so- 
ciety institutions. 

The Hamiltonians include traditional centrists (like Rob Atkinson, who re- 
cently co-authored Big is Beautiful with Michael Lind), as well as voices on both 
ends of the political spectrum. Recapitulating Schumpeter’s praise of monopoly 
as a spur to growth, Peter Thiel’s Zero to One is a paean to monopoly power, justi- 
fying its perquisites as the just and necessary reward for dramatic innovation. On 
the left, Evgeny Morozov does not want to see the data stores of the likes of Google 
and Facebook scattered to a dozen different versions of these services. Rather, he 
argues, they are most likely natural monopolies: they get better and better at each 
task they take on when they have access to more and more pooled data from all the 
tasks they perform. The ultimate left logic here is toward fully automated luxury 
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communism, where massive firms use machine learning and 3-D printing to solve 
hunger, save the environment, and end the problem of scarcity.” Left centralizers 
also argue that problems as massive as climate change can only be solved by a 
Hamiltonian approach. 

The Jeffersonian and Hamiltonian visions lead to very different policy rec- 
ommendations in the tech space. Jeffersonians want to end Google’s acquisition 
spree, full stop. They believe the firm has simply gotten too powerful. But even 
some progressive regulators might wave through Google’s purchase of Waze (the 
traffic monitoring app), however much it strengthens Google’s power over the 
mapping space, in hopes that the driving data may accelerate its development of 
self-driving cars. The price of faster progress may be the further concentration of 
power in Silicon Valley. To Jeffersonians, though, it is that very concentration (of 
power, patents, and profits) in megafirms that deters small businesses from tak- 
ing risks to develop breakthrough technologies. 

Facebook’s dominance in social networking raises similar concerns. Privacy 
regulators in the US and Europe are investigating whether Facebook did enough 
to protect user data from third-party apps, like the ones that Cambridge Analytica 
and its allies used to harvest data on tens of millions of unsuspecting Facebook us- 
ers. Note that Facebook itself clamped down on third party access to data it gath- 
ered in 2013, in part thanks to its worries that other firms were able to construct 
lesser, but still powerful, versions of its famous “social graph”—the database of in- 
tentions and connections that makes the social network so valuable to advertisers. 

For Jeffersonians, the Facebook crackdown on data flows to outside develop- 
ers is suspicious. It looks like the social network is trying to monopolize a data 
hoard that could prove essential raw materials for future start-ups. However, 
from a Hamiltonian perspective, securing the data trove in one massive firm 
looks like the responsible thing to do (as long as the firm is well-regulated). Once 
the data is permanently transferred from Facebook to other companies, it may be 
practically very hard to assure that it is not misused. Competitors (or “frenemies,” 
in Ariel Ezrachi and Maurice Stucke’s terms) cannot access data that is secure in 
Facebook’s servers—but neither can hackers, blackmailers, or shadowy data bro- 
kers specialized in military grade psy-ops. To stop “runaway data” from creating a 
full-disclosure dystopia for all of us, “security feudalism” seems necessary. 

Policy conflict between Jeffersonians and Hamiltonians, “small is beautiful” 
democratizers and centralist bureaucratizers, will heat up in coming years. To un- 
derstand the role of each tendency in the digital sphere, we should consider their 
approaches in more detail. 


2 Authors in this vein include Leigh Phillips and Michal Rozworski, People’s Republic of Walmart (Ver- 
so, 2019); Aaron Bastani, Fully Automated Luxury Communism (Verso, 2019), and Peter Frase, Four 
Futures (2016). 
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The Jeffersonian Critique of Absentee Ownership 


The largest, most successful firms of digital capitalism tend to serve as platforms, 
ranking and rating other entities rather than directly providing goods and ser- 
vices. This strategy enables the platform to outsource risk to vendors and con- 
sumers, while it reliably collects a cut from each transaction. Just as a financial 
intermediary may profit from transaction fees, regardless of whether particular 
investments soar or sour, the platform pockets revenues on the front end, regard- 
less of the quality of the relationships it brokers. 

This intermediary role creates numerous opportunities for platforms. For ex- 
ample, they police transactions and adjudicate disputes that used to be the pre- 
serve of governments. I call this powerful new role of platforms “functional sov- 
ereignty,” to denote the level of power a private firm reaches when it is no longer 
one of many market participants, but instead, the main supervisor and organizer 
of actual market participants. Platforms like Amazon and Google are functionally 
sovereign over more and more markets, playing a quasi-governmental role as they 
adjudicate conflicts between consumers, marketers, content providers, and an 
expanding array of third and fourth parties. 

Personalization is a mantra for digital strategists, who tend to assume it is a 

“win-win” proposition. For example, tailored search results both guard Google’s 
users against distraction and tend to connect them to products they want. How- 
ever, online markets premised on ever greater knowledge of our desires and “pain 
points,” income level and wealth, can easily tip toward exploitation. Platforms 
have an interest in intensively monitoring and shaping certain digital spheres in 
order to maximize their profits (and, secondarily, to maintain their own repu- 
tations). However, in their ceaseless quest to annex ever more sectors into their 
own ecosystems, they all too often bite off more than they can chew. They tend 
to overestimate the power of automation to process all the demands that modern 
marketplaces generate. 

This has led to another problem, familiar from the history of monopolistic 
enterprise: absentee ownership. When a massive firm buys a store thousands of 
miles away from its headquarters, it owns the store, and will seek profit from it, 
but it may only assess its performance in crude terms, with little interest in the 
community in which the store is embedded. The store may neglect traditional 
functions it served, simply in order to maximize the revenues that its absentee 
owner demands. A present owner, resident in the community, is more likely to 
run the store in a way that comports with community interests and values, since 
the present owner will itself experience any improvement or deterioration the 
store causes in its community. 

Similar dynamics emerge online. Google owns the largest collection of videos 
online, but its YouTube subsidiary’s profitability depends on calculated neglect 


289 


290 


Frank Pasquale 


of many aspects of the platform. Over the past two years, a litany of critics have 
flayed the firm for promoting disturbing, tasteless, shocking, and abusive content, 
even to children. The recent Google announcement that it would promote Wikipe- 
dia links to debunk the conspiracy theory videos YouTube does so much to pro- 
mote, represents yet more layers of outsourcing—from a for-profit corporation to 
a non-profit that in turn delegates power over content to volunteers managed by a 
shadowy layer of administrators. 

For Jeffersonians, the answer here is obvious: there should not be one, behe- 
moth corporation with power over so many videos. YouTube says it needs the scale 
to keep its offerings free; Jeffersonians respond that the ad-driven business model 
is just a way to undercut subscription services which could better manage their 
offerings. Jeffersonians also point out that it is very difficult to know the extent to 
which services like YouTube are actually serving users and content producers, and 
to what extent they exist simply to maximize ad revenue. 


A Hamiltonian Perspective on New Digital Utilities 


The guiding spirit of Jeffersonians is the original intent of U.S. antitrust law—that 
immense corporations were so capable of dominating their customers, employees, 
and communities, that they needed to be broken apart. Dividing a large corpora- 
tion into smaller part is a “structural remedy,” because it addresses fundamental 
ownership stakes and control in society. This populist demand to break up the 
largest corporations has inspired antitrust attacks on firms ranging from Stan- 
dard Oil to Brown Shoe to Microsoft. 

More recently, though, antitrust authorities have been more cautious about 
breaking up large firms. Both the Department of Justice and the Federal Trade 
Commission have narrowed their interest to focus almost entirely on large firms’ 
present, price effects on consumers. So a massive firm that undercuts competi- 
tors by reducing quality is of little concern to them. Nor is the possibility that the 
same firm will, eventually, once it has monopolized a space, raise prices dramati- 
cally for customers (or reduce wages for workers). Instead, there is a single-mind- 
ed devotion to efficiency—more, for less, faster. Free or low prices in the short run 
trump other considerations. 

To see the practical effects of this obsession with the short-term, imagine 
searching for “weather” in Google, and instantly seeing its own weather forecast 
filling your mobile screen. Had it linked to three forecasting sites in that pre- 
cious screen space, it might have directed more exposure and advertising reve- 
nue to sites with diverse interfaces, more or less information, or other variations. 
For example, the site WeatherSpark used to give a beautifully precise image of 
storms’ movement over time—the perfect visual analogue to Accuweather’s min- 
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ute-by-minute forecasts of rain or clear skies. But WeatherSpark no longer offers 
that service, and who knows how many other startups gave up on occupying this 
space. To establishment antitrust authorities, there is no ground to intervene— 
consumers get the basics of weather from Google’s interface, and it is free. It’s a 
short-termist outlook that omits long-run considerations in the name of a presen- 
tist scientism. In their worldview, there is no room for argument about whether 
better or worse alternatives do or should exist. Antirust is supposed to protect 
“competition, not competitors”—and a singular lack of concern for quality trans- 
lates into profound lack of interest in whether current or future competitors could 
do a better job than a digital behemoth. But how can we know if there is competi- 
tion, if there are no competitors to provide it? 

In the wake of this narrowing of antitrust law, more Hamiltonian voices have 
called for a revival of public utility law to cabin the power of massive online firms. 
The utility regulators of the early 20% century did not want to see 10 different 
phone companies digging up the streets to provide competition in calling services. 
Nor did they envision localized power generation (however tempting that pros- 
pect may now be for those pursuing a distributed, renewable grid based on so- 
lar power). Instead, these regulators accepted the massiveness of telecom, power, 
and other firms as an inevitable aspect of modern economic rationalization. They 
just wanted a state (and unions) massive enough to offer countervailing forces. 

For the Hamiltonians, an agency like the Federal Communications Commis- 
sion provides a behavioral alternative to structural remedies. A Federal Search 
Commission, for example, could monitor how Google treats competing firms in 
search results, and force it to provide alternatives to its own services in such re- 
sults.’ European competition authorities may effectively create such an agency, if 
they are serious about policing Google’s treatment of vertical search competitors 
(that is, narrow gage searching for certain types of goods or services). 

Hamiltonians identify with technocratic left-liberalism. They want to deploy 
tools like cost-benefit analysis and advanced data analysis to calculate just when 
it might make sense for a service to be folded into a conglomerate, and when it 
makes sense to create rules that presume the independence of firms. Howev- 
er, there are more ideologically ambitious endorsements of industrial scale and 
scope. For example, Evgeny Morozov warns against efforts to split up Google or 
Facebook, since advances in AI may only be possible when truly massive amounts 
of data are consolidated. In a recent podcast, the socialists of Chapo Trap House 
joked that they were happy to see Amazon consolidate power. Once it takes over 
every business in the country, it will be easy to “cut off the head” and simply im- 
pose government control over the economy. “Free Whole Foods hot bar for every- 


3 O. Bracha & F. Pasquale, Federal Search Commission: Access, Fairness, and Accountability in the Law of 
Search, 93 Cornell L. Rev. 1149 (2008). 
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one!” was the imagined denouement. Similarly, ifall the private health insurers in 
the US merged, the stage would finally be set for “single payer:” the government 
need only take over the one insurer left standing. 

Authors at Jacobin (including Alyssa Battistoni, Peter Frase, Christian Parenti) 
are also articulating a neo-Hamiltonian approach of advanced corporate capaci- 
ty tempered by countervailing power of government and labor unions. Allowing 
centralization into large peak organizations like Germany’s general trades union 
council and mega-manufacturers, would enable corporatist negotiations over 
the division of the spoils from the types of investment made possible by massive 
concentration of resources. Germany’s largest trade union recently negotiated to 
reduce its members’ workweek to 28 hours, while also getting a 4.3% pay raise— 
exactly the type of deal U.S. workers could have gotten had productivity gains 
since the late 1970s been widely shared, and had business and labor been similarly 
organized. 

At its most ambitious, the Hamiltonian vision tends toward a dream of a ro- 
bust universal basic income guaranteed under fully automated luxury commu- 
nism. Artificial intelligence and robots mimic workers, who still are paid for the 
data they (or their forbears) contributed to advance Al’s development. Hamiltoni- 
anism can be the economic equivalent to geoengineering—an embrace of the rad- 
ically new and large-scale, arising out of the sense that inequalities and climate 
change are such massive problems that only rapid technological advance can solve 
them. Jeffersonians adhere to something like a precautionary principle, question- 
ing whether any entity should accumulate the power necessary to, say, compare 
everyone’s genomes, convert millions of workers’ movements into patterns of be- 
havior programmable into robotics, or maintain social credit scores on all citizens. 


Reconciling Jeffersonian and Hamiltonian Perspectives 


All these trends suggest new fault lines in economic thought for the 21“ century. 
To alleviate these tensions, we should return to some seminal tensions in the neo- 
liberal project. In the 1930s and 40s, the University of Chicago economist Henry 
C. Simons warned that monopolies posed a mortal threat to classical liberal ide- 
als of free and open markets. In his A Positive Program for Laissez Faire, written 
in 1934, Simons argued that “the great enemy of democracy is monopoly, in all 
its forms: gigantic corporations, trade associations and other agencies for price 
control, trade-unions—or, in general, organization and concentration of power 
within functional classes.” However, by the 1950s, George Stigler and Aaron Di- 
rector supplanted Simons at Chicago, and offered a far more hands-off approach 
to antitrust law. They viewed concentrated state and union power as a far greater 
threat to society than concentrated corporate power. And since the former was 
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needed to combat the latter, they downplayed the harm that massive corporations 
could pose (outside a narrowly delimited category of conduct that was to become 
ever smaller as Chicago scholars like Robert Bork shrank the domain and force of 
antitrust law). 

What if Chicago had followed Simons’s path instead of Director’s? Neoliber- 
als might have embraced a more even-handed approach to confronting excessive 
power in society. Antitrust authorities would have better resisted behemoth firms’ 
aspirations to centralize data collection and control of workers. Policymakers 
could have better balanced efforts to reduce state power with parallel efforts to 
decrease corporations’ ability to work their will upon communities and workers. 
A 1950s era policy agenda to reduce union power looks risible in the 2010s, when 
union density has declined so precipitously, while corporate concentration has 
risen. 

Jeffersonians have their own blind spot when it comes to labor. Too much of 
the Jeffersonian literature idealizes small-holders, advancing an idea of every- 
man-as-entrepreneur. But most of us are, and will be, working for someone else 
for most of our life. Thus Atkinson and Lind are right to argue, in Big is Beautiful, 
that small businesses should be held to many of the same labor and consumer pro- 
tection laws that now only govern larger corporations. Otherwise, the wizards of 
franchising and platform capitalism will simply find new ways to disaggregate 
existing concerns into smaller units, to get around regulation. Undercapitalized 
and judgment-proof small businesses are the perfect business law-breakers, since 
they have little to lose if caught. 

However, a core insight of the Jeffersonians must be respected: there really 
is no “one best way” to handle many products and services. The question then 
becomes, how to identify optimal scale and scope of enterprise in different in- 
dustries. When a firm has a bona fide need for data to solve a problem (such as 
calculating optimal routes for a fleet of self-driving cars), that is a much better 
rationale for “bigness” than simply using data to rearrange commercial transac- 
tions to its own advantage. Stacy Mitchell of the Institute for Local Self-Reliance 
has observed that, “when third-party sellers post new products, Amazon tracks 
the transactions and then starts selling many of their most popular products.” 
However much that practice increases economic productivity, it does so at an un- 
acceptable cost of concentrating power in one firm while discouraging entrepre- 
neurship outside it. Policymakers should protect vulnerable sellers against it. 

The structural concerns of the Jeffersonians are a first line of defense against 
over-concentration in the economy. Competition authorities should take them se- 
riously, particularly when there is no substantive productive rationale for bigness. 
If Amazon needs to buy equipment manufacturers to pursue vertical integration 
to make a better Kindle, fine—but if it is acquiring other firms simply (or mainly) 
to enhance its bargaining power relative to consumers or suppliers, that is not a le- 
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gitimate rationale for mergers. Similarly, authorities need to recognize that merg- 
ers in the name of “better service” or “cheaper inferences” about users can lead to 
overwhelming bargaining power for a platform vis a vis advertisers it serves—and 
its ability to intrude upon the privacy of its users. Those are the key reasons why 
the FTC should have blocked Google’s purchase of DoubleClick, and Facebook’s 
acquisition of Instagram and WhatsApp.* 

It will be politically difficult to “unscramble the omelet” of currently dominant 
firms. Authorities are wary of reversing mergers and acquisitions, even when they 
are obviously problematic in hindsight. While Jeffersonians may keep our digital 
giants from getting bigger, Hamiltonians will need to monitor their current prac- 
tices, and intervene when they transgress social norms. Thanks to the movement 
for algorithmic accountability, we know that algorithmic corporate decisionmak- 
ing is frequently deployed to arbitrage around extant anti-discrimination, due 
process, and media law. Agencies like the Consumer Financial Protection Bureau, 
the Federal Communications Commission, the Federal Trade Commission, and 
state attorneys general, should closely monitor platforms in order to ensure that 
they are actually giving their users a fair shot at access to customers, advertising, 
and growth. These firms no longer are mere market participants. They make mar- 
kets, and need to be treated as such. Even Mark Zuckerberg recently conceded that 
the question is not whether to regulate Facebook, but how. Other tech CEOs should 
adopt a similar openness to the societal values they have shunned for so long. 


Context Matters 


There is ongoing struggle over what responsibilities the domination of an online 
space should entail. Investors demand a fantasy of monopolization: their firm not 
merely occupying a field, but developing “moats” against entrants, to guarantee 
both present returns and future growth. However, the day-to-day reality of op- 
erational budget constraints pushes the same firms toward the pathologies of ab- 
sentee ownership. 

Law can help resolve these tensions. Competition laws take aim at the function- 
al sovereignty of large tech platforms, reducing the stakes of a firm’s domination 
of a field. At the very least, antitrust authorities should have blocked Facebook’s 
purchases of Instagram and Whatsapp, instead of letting its juggernaut of dom- 
ination over communication roll up some of the few entities capable of providing 
alternative modes of association online. Ten, twenty, or one hundred social net- 


4 For more on the advantages and disadvantages of antitrust policy here, see Frank Pasquale, 
Dominant Search Engines: An Essential Cultural & Political Facility, in The Next Digital Decade 
(2011). 
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works could eventually emerge, if competition law were properly enforced, and in- 
teroperability standards could assure smooth connections among confederations 
of social networks, just as AT&T, T-Mobile, and Verizon customers can all talk to 
one another seamlessly. If that diversity emerged, we could worry less about a few 
persons in Silicon Valley essentially serving as a world Supreme Court deciding 
which expression was appropriate for a so-called “global community,” and what 
should be banned or obscured (in oft-secretive algorithmic manipulation).° 

When industrial giants can’t be broken up, there are still many ways to neu- 
tralize their power. Utility-style regulation mitigates the worst failures of absen- 
tee owners, as well as the caprices of the powerful. The state can require Google to 
carry certain content on YouTube, just as it has required cable networks to include 
local news. Moreover, whenever policymakers are afraid that firms like Google, 
Amazon, or Uber are taking too large a cut of transactions, they can take a page 
out of the playbook of insurance regulators, who often limit insurers to taking 
15% to 20% of premiums (the rest must be spent on health care). That kind of limit 
recognizes the infrastructural quality of these firms’ services. We would not want 
to live in a world where the electric company can endless jack up charges in order 
to take advantage of our dependence on it. Digital monopolists should face similar 
constraints. 

Though Jeffersonian trust-busters and Hamiltonian utility regulators have 
very different views of political economy, each counters the untrammeled aspira- 
tions (and disappointing quotidian reality) of stalwarts of digital capitalism. They 
also help us understand when giant firms can help us solve the “knowledge prob- 
lem” Hayek identified, and when they exacerbate it via obscurity and obfuscation.® 
If conglomeration and vertical mergers actually help solve real-world problems— 
of faster transport, better food, higher-quality health care, and more—author- 
ities should let them proceed. Such industrial bigness helps us understand and 
control the natural world better. But states should block the mere accumulation 
of bargaining power and leverage. Such moves are exercises in controlling per- 
sons—a much less salubrious aim of industrial organization. Economic policy 
focused on productivity and inclusive prosperity will balance and do justice to im- 
portant insights from both Jeffersonian and Hamiltonian critics of our increas- 
ingly sclerotic economy. 


5 Kate Klonickand Thomas Kadri, “How to Make Facebook’s ‘Supreme Court’ Work,” N.Y. Times, Nov. 
17,2018. 


6 Walter Adams and James W. Brock, The Bigness Complex (Stanford University Press, 2004). 
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| want to be amachine 
Andy Warhol 


|. Introduction 


The current hype on Artificial Intelligence is exaggerated insofar, as the much-dis- 
cussed procedures of machine learning adress relatively specialized problems, e.g. 
of image recognition or more generally of pattern discrimination. A universal, 
general artificial intelligence is not the goal and perhaps not even possible. Nev- 
ertheless, there is one related phenomenon that emerges in sometimes nervous 
discussions nowadays: It is the question whether the systems of machine learning 
can be ‘creative’. This discussion heated up as Alpha Go was able to beat Lee Sedol 
in Go—especially the famous move 37 in match two on the 10" of march 2016, 
which seemed radically new even for experienced Go-Players was discussed: No 
one foresaw this move—so could it be considered as creative or not?! For obvious 
reasons, this essay cannot go into the depths of the theory of creativity or ‘creative 
subjectivity”, but it will adress a recent phenomenon in the discourse on the ‘new’ 
AI, a phenomenon, in which the question of creativity is especially urgent: that is 
the field of art, art created by AI. 


1 See the fundamental critique of the idea of machine creativity by Mersch (2019). 


2 Cp.amongst many others Sternberg (1999) and Reckwitz (2012). 
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The AI-Art Gold Rush Is Here 


An artificial-intelligence “artist” got a solo show at a Chelsea gallery. Will it reinvent art, or 
destroy it? 


Al-generated “faceless portraits" by Ahmed Elgammal and AICAN. 


Fig. 1: AI Art Goldrush is here 


The AI Art Gold Rush is here is the title of a critical essay by Ian Bogost (2019). The 


images depicted somehow look like deformed Renaissace portraits, perhaps with 
a little Francis Bacon in it (Fig. 1). 
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Fig. 2: Portrait of Edmond de Belamy 
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Especially the slightly fuzzy character of the portraits resembles the famous Portrait 
of Edmond de Belamy, auctioned for 432.500 $ at Christie’s in october 2018. That this 
computergenerated image was sold for a comparatively high price was surprising, 
the joke with an algorithm as the signature spurred again a discussion on the ‘cre- 
ativity’ of AI and the question emerged if an AI system can be an artist or an author. 
In addition, a discussion began whether the art collective Obvious who used the AI 
system is the ‘real’ author or even the programmer who developed some of the al- 
gorithms.’ But the artwork was surprising also in another respect: It took recourse 
to a quite conservative genre and a quite conservative style of depiction: the blur- 
riness seemed not only to evoke a certain ‘technicity’, but a conservative notion of 
artiness'—see the somewhat weird discussions on the ‘blurred’ style of the impres- 
sionists (cf. Payne 2007). Given that the development of painting in the twentieth 
century developed new forms like abstraction raises the question why one needs a 
conservative style of painting in order to demonstrate the creativity of AI? 

Andeven the joke with the signature shows a certain traditional understanding 
of art, insofar as many artists (see the famous quote of Andy Warhol above) prob- 
lematized the traditional myth of the artist: just think of surrealistic automatic 
writing, Cage’s aleatoric processes or the doubling gestures of Elaine Sturtevant 
to name just a few. Perhaps Obvious understood the painting as purely ironical— 
and intended to ridicule the the ‘newness’ of AI by the very act of foregrounding 
its conservative ‘taste’. Often the ‘democratization’ of new technologies need the 
adaption of conservative and established forms to be adaptable to mass markets. 
Therefore, an abstract work of art might be regarded as insufficient to prove Al’s 
creativity, for the simple reason that many people still have problems in accepting 
abstract art forms as art at all—or as ‘too easy’ for real art. 

Hence and this is the central argument of my short essay it is crucial to his- 
toricize this discussion on ‘Al art’, the implications of ‘machine creativity’, and 
therefore the (possible) automatization of artistic work: Sometimes, we forget that 
the idea of computational machines, or more specifically AI, making art adds toa 
rather nervous discourse on the automation of work through smart machines.’ If 
machines can produce art, the assumption goes, they could mass produce art for 
everyone, serially, industrially. Then there would no longer be auratic works of art 
made by rare geniuses. And nobody would have to pay millions for artworks. 


3 Cp. the insightful article by Sudmann (2019). 

4 Alsystems do not necessarily have to produce blurry images—but ‘modern art’ in a way has to 
transgress notions of realism in a way. 

5 Cp. my article “Digitale Technologien und das Verschwinden der Arbeit” (2019). See also Bo- 
gost (2019): “Given the general fears about robots taking human jobs, it’s understandable that 
some viewers would see an artificial intelligence taking over for visual artists, of all people, as a 
sacrificial canary.” 
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In part II, I want to look specifically on ‘information aesthetics’, a discourse 
and practice from the 1960s, already driven by the idea to produce art by com- 
puters (not mainly with computers as tools).° In part III, I will discuss somehow 
speculative reasons why the idea to automatize artistic work (and hence simply 
mass produce ‘art’ with machines) did not seem to work back then.’ In part IV, I'll 
come back to the recent ‘gold rush’ in ‘Al art’ and re-read it in the light of the dis- 
course on information aesthetics. 


Il. Short Remarks on Information Aesthetics 


The origin of information aesthetics is the attempt to formally determine the 
‘measure’ of aesthetics. In 1933, David Birkhoff formulated an equation (Fig. 3) in 
which the variable O denotes the measure of the order of a given work and the 


variable C the ‘complexity’. M is the degree of how aesthetic an artwork is. 
Fig. 3: Birkhoff-Equation 


According to it, the more ordered and the less complex a work of art is, the more 
aesthetic it would be. Apart from the fact that it is difficult to understand exactly 
how to determine the degree of order and complexity in a specific case, this at- 
tempt to express the ‘aesthetic quality’ of art in an equation seems strange to us 
today. Nonetheless, especially in the 1960s there were several attempts to formally 
understand art and its aesthetic criteria and consequently to produce it syntheti- 
cally with computers (although the computers were slow, the output possibillities 
limited and computer technologies were only available in research institutions 
and large companies). 


6 There are even more precursors, for example the computer cluster lamus, who is composing 
music and even released an album, (see: http://melomics.uma.es/). The slogan is, not surprising, 
“music for everybody, everything“, promising a democratization of art. Another important exam- 
ple would be Aaron, which is a software system that in collaboration with its inventor Harold Co- 
hen produces paintings (see: https://www.computerhistory.org/atchm/harold-cohen-and-aar- 
on-a-40-year-collaboration/). 

7 It’s quite problematic that an otherwise excellent collection of essays on ‚Computers and Cre- 
ativity‘ (McCormack/d’Iverno 2012) does not include a single contribution that tries ot relate the 
central question of the volume to sociological questions concerning the art system. ‘Art History’ 
is only mentioned a few times in the outstanding contribution by Frieder Nake—except of Nake’s 
paper most contributions in this volume de-socialize and de-historicize the question of the pos- 
sible creativity of machines. 
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Fig. 4: Piet Mondrian, Composition With Lines (1917) 


In 1967, Michael Noll describes in his essay “The Computer as a Creative Medium” 
an information aesthetic experiment with a painting by Mondrian (Fig. 9.° Noll 
writes: 


[An] experiment was performed using Piet Mondrian’s “Composition With Lines” 
(1917) and a computer-generated picture composed of pseudorandom elements 
but similar in overall composition to the Mondrian painting. Although Mondrian 
apparently placed the vertical and horizontal bars in his painting in a careful and 
orderly manner, the bars in the computer-generated picture were placed accord- 
ing to a pseudorandom number generator with statistics chosen to approximate 
the bar density, lengths, and widths in the Mondrian painting. Xerographic copies 
of the two pictures were presented, side by side, to 100 subjects with educations 
ranging from high school to postdoctoral; the subjects represented a reasonably 
good sampling of the population at a large scientific research laboratory. They 
were asked which picture they preferred and also which picture of the pair they 
thought was produced by Mondrian. Fifty-nine percent of the subjects preferred 
the computer-generated picture; only 28 percent were able to identify correctly 
the picture produced by Mondrian. In general, these people seemed to associate 


8 It is by the way one of the earliest texts in which the computer is described as a medium—the 
computer’s becoming a medium thus goes back to questions of art and aesthetics. 
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the randomness ofthe computer-generated picture with human creativity where- 
as the orderly bar placement of the Mondrian painting seemed to them machine- 
like. This finding does not, of course, detract from Mondrian’s artistic abilities. 
His painting was, after all, the inspiration for the algorithms used to produce the 
computer-generated picture, and since computers were nonexistent 50 years ago, 
Mondrian could not have had a computer at his disposal. (1967: 92) 


Fig. 5: Simulated Mondrian by Michael Noll 


Noll thus simulates a Mondrian (Fig. 5) on the basis of a statistical distribution 
which is supposed to describe the arrangement of the lines in Mondrian’s work. And 
since this distribution looks more ‘disordered’, a group of observers, presumably not 
to be regarded as representative, identifies the simulated Mondrian as the real one, 
while the real one appears too regular and mechanical.’ Does this preference show 
that the disorderly picture is understood to have a higher aesthetic quality? That 
would contradict Birkhoff, but it does remind us in an uncanny manner of the dis- 


9 A similar discourse can be found in recent Al art: “According to Elgammal, ordinary observers 
can’t tell the difference between an Al-generated image and a ‘normal’ one in the context of a 
gallery or an art fair” (Bogost 2019). 
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torted, blurred, and ‘arty’ qualities of recent Al-generated portraits. Or is it because 
the disorder is simply understood as a reference to human authorship? It could also 
reveal that the artiness of a given picture does not depend on perceptual features 
alone... It should be noted that Noll tries to find formal, algorithmic rules, allowing 
the production of a work of art that can be identified as a work of art. Nollis not the 
only one following this approach. Frieder Nake, one of the most important theorists 
and practitioners of information aesthetics, also tried to trace the pattern of image 
production in Paul Klee’s work and thus produce a computer-generated Klee (Fig. 6). 
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Fig. 6: Simulated Klee by Frieder Nake 


Admittedly, these examples are taken out of context and an appropriate discus- 
sion of information aesthetics would have to consider at least the positions of Max 
Bense, Rul Günzenhäuser, and Abraham Moles, which can’t be addressed in this 
essay. In any case, from today’s point of view, these attempts seem rather strange, 
because they remove the works of art from their historical context and reduce them 
to abstract structures that can be formalized—similar to recent Al-produced art. 
It must be therefore emphasized that the aesthetic strategies of Mondrian and 
Klee have reacted to certain historical questions, not the least of them the history 
of art itself. Information aesthetics and AI art seem to understand artworks as 
ahistorical, formal structures—although they could also be seen as responses to 
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a certain historical, in this case computational, context of art as such (cf. to the 
vexed relation between form and history amongst others Buchloh 2015). This also 
seems to reappear in recent Al art: “That might be an inevitability of AI art: Wide 
swaths of art-historical context are abstracted into general, visual patterns.” (Bo- 
gost 2019)"° 

However, these attempts are relatively characteristic of a larger development, 
namely the attempts to formalize cognitive labor and, if possible, to transfer it 
partially or entirely to machines, for which computers as symbol-processing ma- 
chines are suitable." Early texts from the domain of Computer Science, such as 
Douglas Engelbart’s “Program on Human Effectiveness” or J. C. R. Licklider’s 
“Man-Machine Symbiosis” from the 1960s, are programmatic for this. (cf. Licklid- 
er 1960: 4-11; Engelbart 1991: 235-244). Noll, Nake and others try to make aesthetic 
work formalizable and, in principle, executable by machines. Their attempts can 
be understood, whether intentionally or not, as contributions to the automation 
of aesthetic work. 

In other forms of commodity production, these processes normally contribut- 
ed to the cheapening of commodities—and therefore to ‘democratization’ in the 
sense of goods becoming more affordable for more people. That was obviously not 
the goal of information aesthetics, since—for instance—Nake operated as the au- 
thor of the work (and not Klee); and if he would have called his work a Klee, this 
could have led to serious legal problems. But even if this democratization of art 
would have been the goal of information aesthetics—to start an industrial, cheap 
mass production of, say, Mondrians and Klees—this would not have worked in 
the art system: Art is not only a question of formal structures and strategies, it 
is also a question of historical and especially social places and roles. We will now 
turn to that. 


Il. Art, knowledge and work 


Apparently, art it is not directly threatened by computerization. Art does not 
appear in the highly discussed Oxford research report, which started a nervous 
discussion on the disappearance of work: the only activity that resembles artis- 
tic practice is that of the ‘art director’, who gets off quite lightly with a 95" place 
on the computerizability probability list. (cf. Frey/Osborne 2013: 59) Artists can- 


10 The same is true for Schmidhubers contribution to McCormack/d’lverno (2012). It’s again an 
approach to formalize aesthetic value of an object—without really posing the question if this 
value is not derived from the relative historical position of the artefact and not (only) of its for- 
mal, internal structure. 


11 Cp. onthe history of Automation Noble (1984). 
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not become the object of rationalization and their works will continue to be called 
‘works’. Artistic work seems to be a type of work that cannot or should not be 
formalized, algorithmized, and consequently neither reproduced mechanically 
without further ado. This suggests the suspicion that artistic work is not really 
work, but another form of activity or—at least—another form of work. But why? 

At least at first glance, the art market looks exactly like any other market: art- 
ists are well aware that they have to earn money with their work. Some succeed 
to an almost inconceivable degree, but most have serious difficulties competing 
in the art market. Artists are also subject to the fact that work is “a necessity of 
refinancing their expenses” (Luhmann 1994: 191). Work, money for one’s own work, 
market, competition, rich versus poor—this first glimpse suggests that the art 
business does not differ in any way from other forms of production. It is not a 
realm of freedom, but only a kind of service or consumer goods industry that 
serves a special market. 

Therefore, we can find cases where work is technologically made superfluos 
in the art business. If one looks at a large studio, such as Studio Olafur Eliasson, 
one could observe that the introduction of new computer-assisted technologies 
directly leads to the disappearance of work. A series of jobs, e.g. website main- 
tenance, management, public relations, logistics, up to the people that clean the 
studio, may be replaced. Inasmuch as Studio Olafur Eliasson also operates under 
capitalist conditions, it is likely to save costs by rationalizing the way work is done. 
However, this aspect is external; it does adress the problem that information and 
generative aesthetics have pointed to, mainly if the work of making art itself can 
be rationalized. Although Olafur Eliasson could not realize any of his elaborate 
projects without his team, the disappearance of the teamwork in the black box of 
the author’s name does not seem to affect art ‘in itself. And insofar the teamwork 
is blackboxed it may change its composition without changing anything in the 
‘artiness’of Eliasson’s art. 

This finally brings us to the core of the problem raised by information aesthet- 
ics. Obviously, the idea of art without the intervention of a human author or caus- 
er—and even if its role consists precisely in demonstratively withdrawing—does 
not seem plausible to us. We do not see art appearing in nature.” The question 
implied in information aesthetics and Al art is different from the observations in 
Walter Benjamin’s famous essay Das Kunstwerk im Zeitalter seiner technischen Re- 
produzierbarkeit, where everything revolves around the question of the technical 
reproduction of the product, the work of art. But in information aesthetics and AI 
art the question of the reproducibility of the work that produces the work of art is cen- 
tral. The fact that the technical reproduction of this work does not seem possible 


12 Although art was historically sometimes understood as being close to nature (‘Kunstschönes’ 
vs. ‘Naturschönes’), cp. on this Kant (1914 [1790]). 
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does not have to be attributed—this would be a very traditional answer—to the 
‘genius’, which is ultimately of divine origin and thus per se untechnical. Luhmann 
remarks: “The artist’s genius is primarily his body” (2000: 38). One could therefore 
simply say that the separation of knowledge from the working body, characteristic 
of the progression of capitalism and perhaps first discovered by Marx, does not or 
cannot take place in art. But why? 

One possible reason would be that the work of art—despite the attempts of 
the information aesthetics (at least according to Birkhoff) to formalize precisely 
the complexity—is too complex and thus the work that produces it cannot be suf- 
ficiently understood. Thus, it is noticeable that Noll and Nake focus on a certain 
type of painting, which is determined by the extensive recourse to basic geomet- 
ric forms. Such forms seem simple enough to suggest their formalization, while 
other more complex ones would elude them. Obviously, technology progressed: 
Nowadays geometric patterns do not suffice to demonstrate the creativity of com- 
puting systems, it has to be (albeit blurry) portraits. Portraiture is historically con- 
nected to a history of ‘genius’. As was remarked in the beginning, such portraits 
are hardly the status quo of art today'’—using machines to create figurative art 
is less about Als critically reflecting on contemporary art (at least when Nake and 
Noll made their works, abstract painting was quite central), it is more about the 
new powers of computer graphics. 

But coming back to the vexed question of auctoriality: Even with geometric 
forms, it is true that we cannot quite imagine their existence as art without an art- 
ist. For even if an artist—like Nake, for example—were to define himself precisely 
by delegating all work to machines, we would still call the result ‘a work by Nake’, 
which is similar to the finding already mentioned that the production within a 
studio with a division of labour is ‘black boxed’ under an author‘s name. A similar 
process seems to occur with the debate regarding the Portrait of Edmond de Belamy 
on who is the author in the last instance. 

The work of art is the result of a work in which body and knowledge, i.e. the 
knowledge of how this specific work is to be produced, cannot be separated—and that 
means that the function of the ‘author’ is central. (cf. Graw 2012: 43-45) At least, 
this is the ideological figure that has historically emerged as characteristic of the 
art system. Therefore, works of art must not be the connection between knowl- 
edge and a false body—this would be what we call forgery. Even if, for example, 
one were to take a work by Donald Judd that according to Sebastian Egenhofer 
(2008: 214) is “dissolved in the anonymity of the industrial dispositive” , it would 
still be pointless if another person or simply a company, based on the knowledge 
of how it is made, were to produce the same object again, as it is also done in 
principle in the industrial production of reproductions—it would not be possible 


13 Perhaps with the exception of some types of ‘post-modernist’ neo-figurative painting. 
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to recognise this reproduction as a work of art (of ‘Donald Judd’). Or let us take 
another example: Elaine Sturtevant borrowed the screen printing matrices from 
Warhol for the Flowers and printed the Flowers again, in 1991 even made an entire 
exhibition with Warhol Flowers—and Warhol, referring to the production process 
of the Flowers, is frequently quoted for saying: “I don’t know. Ask Elaine.” (Quoted 
in: Arning 1989: 44) Nevertheless, Sturtevant’s appropriation of Warhol’s knowl- 
edge is not a rationalization of Warhol’s work in the sense that Sturtevant now 
simply makes ‘cheaper Warhols’, but she rather makes ‘Sturtevants’. Diedrich 
Diederichsen points out that the work of assigning relevance by curators, critics, 
audiences, etc. also belongs to the work that creates the artwork and its market 
value. (cf. Diederichsen 2012: 99) The ‘distributed’ character of this work makes it 
impossible to rationalize—since it continuously accompanies the work, i.e. never 
ends, and can also take unpredictable turns in the future. ‘Sturtevants’ can be- 
come more important and more expensive than ‘Warhols’ in the future. It could 
even be argued, that Sturtevant by her appropriation makes the author-function 
of Warhol visible in the first place. 

Of course, you can try to save production costs, but it does not make sense to 
offer Warhol’s Flowers cheaper, because only Flowers from Warhol’s Factory are ac- 
cepted as originals, which of course does not exclude the possibility of producing 
inexpensive reproductions of Flowers as posters (which do not count as work of art, 
but its reproduction). Warhol’s life ended in 1987 and that stopped the production 
of original ‘Warhols’—and that’s a necessity: In the long run, the mortality of art- 
ists makes artworks scarce and that’s why they have market-value (cf. critically on 
the notion of scarcity Panayotakis 2012). In capitalism, no one can have an interest 
in a ‘democratic’ production of artworks. 

Obviously, the crucial difference is that in art itself, reproductions, such as 
Sherrie Levine’s re-photographs or Sturtevant’s repetitions, are always originals 
(which, through repetition, exhibit the discourse of the original). In contrast, in 
‘conventional’ commodity production there are no originals," but only series of re- 
productions, e.g. of Flowers posters, which all refer to the original pieces of art but 
are all serially of equal rank. And once again: to be original means to be connected 
with the body of the artist. This work cannot be detached from the body, which is why it 
cannot be formalized and rationalized. The attempts of information aesthetics may 
seem pointless because here a rationalization dispositive from industrial produc- 
tion is transferred to an area that blocks it. Or as Adorno puts it: 


14 Adifficult case seems to be product counterfeiting, insofar as the term already implies an ‘orig- 
inal product’—but even original products exist as a series of identical products. Product coun- 
terfeiting means that a product wrongly claims to belong to that series. This differs from the 
relationship between original and copy in art. 
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On the other hand, however, whenever autonomous art has seriously set out to 
absorb industrial processes, they have remained external to it. [...] The radical in- 
dustrialization of art, its undiminished adaptation to the achieved technical stan- 
dards, collides with what in art resists integration. (1997 [1970]): 217). 


But to be clear about this: That does not mean that art in itself is an utopian realm 
of freedom, freed from the restrictions and rules of commodity production. It just 
means that it is a field of commodity production with different, historical, contin- 
gent rules and that’s why art is in a way economically exceptional (cf. Beech 2015) 


IV. Conclusion 


It seems that the question whether machines can be creative or not can not be 
answered ontologically. What ‘creative’ means seems to be too historically contin- 
gent and malleable. Think of the statements why machine-learning is not creative: 
Bogost (2019) insists that ‘any machine-learning technique has to base its work on 
a specific training set.” That’s true—but doesn’t that also apply to human artists 
who have to train their perceptual and e.g. painterly skills? Or, again Bogost (ibid.): 
“A neural net couldn’t infer anything about the particular symbolic trappings of 
the Renaissance or antiquity—unless it was taught to, and that wouldn’t happen 
just by showing it lots of portraits.” Is that not true for humans too? And more- 
over: Is ‘creativity’ not always distributed between human and non-human actors? 
Don’t human artists often say that their artwork ‘answers’ while they are in the 
painting process? Can one be an artist without any kind of non-human materi- 
als and mediators that are not only transparent tools for a pre-given ‘vision’ of a 
work (cp. Hensel/Schröter 2012)? And couldn’t we even imagine advanced Als, per- 
haps in the shape of humanoid robots that could be artists (cp. Kjgsen 2012 for a 
similar argument relating to the labour theory of value)? Could not the works they 
produce then be tied in the same way to their ‘bodily’ presence as is the case with 
human artists? Think of virtual popstars like Miku Hatsune”, which can operate 
as a kind of enunciator, having a kind of ‘signature’ (here: her voice)—but of course 
the mortality of the artist as a kind of natural scarcity that limits the work is not 
given in sucha case (on the notion of a virtual star, see Schröter 2000). 

Although at the moment the idea of artworks produced by machines seems 
ahistorical and absurd, because artworks have a place in history and have to be 
tied to the body of the artist and therefore be scarce and so on, it may neverthe- 
less be that in a far future things could change: Perhaps a democratization of art 
needs a different social context—as does the democratization of Al, ripping it out 


15 https://en.wikipedia.org/wiki/Hatsune_Miku. 


Artificial Intelligence and the Democratization of Art 


of the hands of big monopolies. That’s why it seems so appropriate that the much 
discussed Al-painting is a portrait of Edward Bellamy—since Bellamy wrote in 
1888 the famous novel Looking Backward 2000-1887. The AI looked back, so to speak, 
to a historical mode of painting in a historical style and with somewhat dated ges- 
tures (the signature), but this might also be a metaphor of looking back from a 
possible future, in which AI can be an artist. Bellamys novel centrally is about a 
very different economy of the future, a kind of post-capitalism, that might indeed 
be the precondition for the democratization of AI as of art. And in the novel sever- 
al futuristic media are mentioned, which nowadays seem absurd of course—but 
which can also be read as metaphors of a future mediality, in which even ‘creativ- 
ity might at least partially be automatized (for an opposing view, see Kelly 2019). 
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“That is a 1984 Orwellian future at our doorstep, right?” 
Natural Language Processing, Artificial Neural Networks 
and the Politics of (Democratizing) Al 


Andreas Sudmann in conversation with Alexander Waibel, professor for Computer 
Science at the Karlsruhe Institute of Technology and also professor at the School of 
Computer Science at Carnegie Mellon University. 


Andreas Sudmann: Alex, you are one of the pioneers in the area of Artificial Neural Net- 
works (ANN) and Natural Language Processing (NLP). What was your initial motivation 
to enter this field of research? 


Alexander Waibel: I have been working in academia for around forty years. Born 
in Germany and having German parents, I went to study at MIT and later to Car- 
negie Mellon, the leading universities in computer science and AI. And it was at 
those institutions that I developed the main thrust and inspiration for all my work, 
which is the question of how we learn and communicate as human beings and how 
we can build technology to help improve human communication. Back then, in the 
1970s, when I went to university, people had already been thinking about AI, given 
that the first definitions of the field were proposed in the 1950s. In fact, Nobel 
laureate Herbert Simon, who was part of my thesis committee, had participat- 
ed in the famous Dartmouth conference on building intelligent machines in 1956, 
where he and other researchers defined this early vision of Al. 

In those days, everybody was attempting to build intelligent machines by 
search algorithms, rules, and logic formulas. But for me as a student, this seemed 
a bit like “des Kaisers neue Kleider” (the emperor’s new clothes). I was listening to 
these famous people talking about a problem that to me would never be solvable 
with a rule-based approach to AI. It was intuitively clear to me that the amount 
of knowledge and facts that we learn in a lifetime is just so enormous that pro- 
gramming it all into rules would be impossible. And worse, they would have to 
be changed all the time, because the world around us is changing all the time. In 
fact, this is totally impossible, and so it was an early concern for me to say from the 
start: We will never achieve such goals unless we develop learning machines that 
can acquire such knowledge by themselves. 
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Aside from this fundamental scientific quest, though, another dimension always 
mattered to me as a scientist: Even though this might now sound like a cliché - it 
was a continuing desire to make contributions to society and to make the world a 
better place, as opposed to just following my own personal pursuits or interests. 
As a researcher, I am not so much simply curiosity-driven, but driven by practical 
goals. Practical goals provide a way to evaluate progress and can impact society 
in a positive way, once we are successful. And among them perhaps one of the 
most consistent goals for my work as a foreigner who grew up with 5 languages, 
was to build machines that can help us humans to translate between languages, by 
text or by spoken language. Throughout history, there have been many attempts 
to use machines for translating texts, which is hard enough in its own right. But 
when you try to connect people across language barriers, you also have to translate 
spoken language. In the 70’s this seemed like a preposterous goal, and indeed, it 
seemed unsolvable, as speech added a whole new dimension of complexity and 
complication to the problem due to the fact that turning speech into text (before 
translating it) was an unsolved AI problem in itself. We did not know how to 
recognize speech and worse people never speak clean text... they make mistakes 
when they speak, they stutter, hesitate and correct themselves during speaking. 
And how would you then combine it with the other hard AI problem of transla- 
tion? And all of that combined was so hard that it was unthinkable to realize in 
the early days of AI. But for me the dream was born and with youthful naivety and 
optimism we went for it. Needless to say, it was and remains a hard problem, a 
problem that we are still working to this very day. But despite the obstacles, chal- 
lenges and delays along the way, we were able to see the fruits of our efforts. In 
retrospect, it is quite a privilege, actually, to be living in the one generation of hu- 
mankind that sees language barriers disappear and to have had the opportunity 
to be working on the technologies made it possible. 

The key to success scientifically was due to progress in machine learning 
methods combined with the explosive growth in available computing power and 
data that supports them. But for the vision to become reality also meant that ac- 
ademic progress had to be transferred to societal deployment. To do so, we start- 
ed several companies that specialized on building wearable speech and language 
technology and eventually mobile speech translators. One of them, Jibbigo, built 
and sold the first ever Dialog translator on a phone. It was sold via the App Store 
and helped Tourists and Healthcare workers to communicate. The company was 
later acquired by Facebook, and we continued working on even more advanced 
deployments. For example, we are now developing new interpreting tools that 
help migrants in Germany to communicate with doctors if they cannot speak the 
language. In a University setting, we have installed automatic simultaneous in- 
terpretations services at KIT, so that foreign students can study in Germany and 
follow a German lecture by way of simultaneous interpretation during the lecture. 
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My team in Karlsruhe and I have also performed early experiments at the Euro- 
pean Parliament to see if such a technology can be of assistance in this most chal- 
lenging language environment. So it’s really ultimately not only about translation 
alone but about how we can build technology that can bridge across barriers, that 
can bring the world together and make people understand each other better. And 
in order to master this hard problem, you really have to build machines that can 
learn effectively at multiple different levels. 


And you have already worked out the necessary fundamentals in the 1980s and 1990s, espe- 
cially with your research on so-called TDNN models. Perhaps you can tell us a little about 
how you came up with this particular approach and explain how it works? 

Back while I was writing my PhD thesis at Carnegie Mellon I became fascinat- 
ed with the idea of building learning algorithms that would mimic more closely 
the massively parallel, holistic learning that we perform as humans in the brain. 
I discovered that researchers in the 1950’s had already proposed so-called “per- 
ceptrons”, which did learn, but could only solve very simple classification tasks. 
Still, the fact that one could actually learn those functions was not only exciting, 
but seemed to be directly applicable to the fuzzy and ambiguous language and 
perception problems that I was working on. Again, this was a time when people 
believed they could solve speech recognition and language translation primarily 
by rules, a belief that seemed preposterous to me, given the enormity of facts and 
details that would have to be assembled. Nevertheless, simple perceptrons and 
similar methods also had severe limitations for speech recognition, because one 
could only train a single neuron at a time. The whole magic of the brain, by con- 
trast is that it does not train single neurons but it trains entire networks of neurons, 
and that an ensemble of neurons can do much more powerful tasks. But how would 
we train an entire network? 

It was just during that time that fortuitously a young assistant professor by 
the name of Geoffrey Hinton came to Carnegie Mellon, and started working on 
something called Boltzmann machines. While he was there, we had many won- 
derful discussions, and he introduced me to something they had been tinkering 
with at USC San Diego, an algorithm called backpropagation. It was much closer 
to what I was looking for and I immediately jumped on it. Backpropagation was 
a simple algorithm, an extension of the simple perceptron - except that this algo- 
rithm would now optimize the whole network of perceptrons and make sure that 
it was functioning in an optimal way. If you tell the entire network what it’s sup- 
posed to do, it can in fact adjust each internal neurons in such a way that each of 
them will try to contribute to what is best for the whole ensemble of neurons. This 
seemed like a big step in the right direction, a big improvement toward classify- 
ing patterns, but for speech and language this was not enough. Because in most 
real world problems, recognizing patterns is not the only problem, but finding the 
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pattern that is to be classified in the first place. This always meant that one would 
have to segment a signal first to find the interesting patterns (sounds, images) 
before they could be classified. In speech, one would have to cut speech in such 
a way that you identify the beginning and the end of a particular phoneme, and 
then once you have that, you can try to apply a neural network for classifying these 
sounds and assemble them into a speech sequence. However, this meant com- 
pounding multiple separate hard problems. And the hard learn lesson in speech 
was that this is deadly as each of them makes mistakes. Therefore, it became clear 
to me that we needed a neural network that was not only a wonderful classifier but 
that would also recognize patterns independent of position, a property we would 
call “shift invariance”. So what does that mean? It means that you are building 
a neural network that you do not just apply to a particular pattern, but that you 
move all neurons over a range of input, and let them essentially scan that input un- 
til it lights up whenever it finds a useful, helpful pattern. Networks of such units 
could thus learn to assemble all useful evident independent of small shifts in the 
signal. Such shift-invariance is, of course, necessary for speech, because speech 
flows by and changes all the time, but as it turns out it is also necessary for many 
other problems in AI, including images, music, games, language, and many more. 

In all of these situations, your first challenge is to know where the useful pat- 
terns are before you can classify them correctly. Hence, classifying things by de- 
tecting them in a shift-invariant fashion was the key problem that we needed to 
solve. With that goal in mind, I then went to Japan as a post-doc, where I had 
access to some of the most powerful super-computers at that time. And with this 
computing power, I had the chance to develop a new model which then became 
known as the time-delay neural network (TDNN). It was still a multilayered 
(“deep”) neural network, but it was now trained specifically for shift invariant 
classification. And as it turned out it worked fantastically well; it worked better 
than all other methods that existed back then. 


So did this new TDNN model then replace other methods? 

Sadly, we still did not have the necessary computing power to build networks that 

were large enough. Back then in Japan, we used the biggest supercomputers avail- 
able, and compared them with other statistical or rule-based methods over bench- 
mark data - and we found them to be much, much better. But when we tried to 

build larger networks and practical speech recognition systems, we still ran up 

against computational limits and had to make many compromises that hurt per- 
formance, and so other researchers could use simpler methods to gradually catch 

up, and get similar or even slightly better performances then we did. As a conse- 
quence - and this was in the late 1990s, early 2000s - people lost interest in neural 

networks and simply used other statistical methods. 
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Ten years passed and few people continued to work on neural networks, until 
around 2008, when - rather by coincidence — various people in the US actually 
tried these old neural networks again, but with the help of much more comput- 
ing power and with much larger amounts of learning data that is now available 
over the internet. And, as it turned out, these neural network methods that had 
already been developed in the 1980s suddenly worked amazingly better than any 
other approach in the field of AI. And they did not just work a little better, but in 
fact they worked like 30 percent better. In our area, you know, entire PhD theses 
are written when progress of half a percent is made; so doing something that is 30 
percent better is simply revolutionary. Asa result, the entire community switched 
to neural approaches within two years. The other thing that came as a surprise is 
what happens when you add more layers in the network. In the 1980s, we had one 
or two so-called hidden layers and that was all we could compute. But now, with 
all this new computing power, we can do three, four, five, and more layers. No 
one expected that this would continue to improve performance, but it did. Today, 
we have networks used for speech recognition in our laboratory with 40, 80 or 
even hundreds of layers. And the exciting neural models that worked so well 20 
years ago work even better today, too. TDNN’s went on and got applied to image 
processing, games, speech, and other problems and became known by the more 
generic name: “Convolutional Neural Nets”. They can now be found at the heart of 
most modern AI engines. 


In terms of having access to powerful hardware and large amounts of data, it was certainly 
helpful that you worked for Facebook for some time. 

Right, I was with Facebook for two years as a director, but of course I also have 
many friends working at Google, Amazon and Microsoft. Many of our students 
are now with Google, Amazon, Microsoft and so on. And many of them graduated 
from our labs. The massive amounts of data that these companies control is of 
course a treasure trove for learning programs. In those large Internet companies, 
they train huge neural networks over huge amounts of data using huge amounts 
of computing power, and the performance gains still grow. And that’s surprising 
and impressive. But if you ask me what’s the new breakthrough in AI today as op- 
posed to 20 years ago, I would have to tell you: not that much. They are again very 
much the same network techniques and training algorithms as we were working 
on in the 1980s, except that we now use orders of magnitude more data and more 
computing power, and they actually work much, much better than we ever imag- 
ined. 
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So far, we have mainly talked about the technological aspects of speech recognition, machine 
translation, and ANN. Perhaps we can now talk about the political dimension of these mod- 
els and applications of AI. What would you consider to be the most relevant political aspects 
concerning the field of natural language processing in general and speech recognition and 
machine translation in particular? 

There is of course much to say about this. One important aspect would be the pol- 
itics of research funding that affects us directly in terms of how we are doing sci- 
ence. Scientific support depends on political factors, and sometimes they work, 
sometimes they don’t. And there are really fascinating differences between the 
US and Germany, or between countries in Europe and Asia, because each of these 
countries or cultures approaches scientific support differently and therefore has 
specific strengths and weaknesses. So I am politically active at that level to intro- 
duce and improve better mechanisms for research support in Europe. The other 
political dimension, though, is what we do with our research. As I have mentioned 
before, in my view of the world, I like to do projects with which I try to improve 
some aspect of society. And as I said before, research always meant for me to make 
people understand each other better. If with our research we can build machines 
that allow us to communicate better, then this means having fewer misunder- 
standings. 

Throughout my career, I have founded several companies, and one of them 
was for building a handheld speech translator on a phone. It was the first mobile 
speech translation system on a phone ever. We launched that in 2009. The start-up 
company was called “Mobile Technologies” and the product was called “Jibbigo”. 
You could speak into the phone and then the system translated the input into an- 
other language. It was a huge success. Apple, for example, ran commercials with 
it. It was used everywhere and people came back to us, saying: “I can finally under- 
stand my in-laws!”, and “I can really understand other people!” And we also start- 
ed doing humanitarian projects, for example, we built systems, say, in Thai and 
Khmer, so that American, European, or Japanese doctors could help rural people 
get healthcare, and we deployed similar things in South America. 

Due to broad interest in this type of technology, the company was then ac- 
quired by Facebook (making the world “open and connected”) in 2013 and for two 
years I led a team of scientists to build translation technology there. At Facebook, 
the use of the technology for translation of posts and other company use cases, 
however, turned out to be of higher priority than the interactive communication 
aspect of our speech translators I was keen on advancing, and so I returned to the 
University to continue our work on the educational and humanitarian aspects of 
this technology. 
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Due to Cambridge Analytica and other scandals, Facebook has increasingly been confront- 
ed with massive criticism, which is why the tech giant is all the more under pressure to meet 

their idealistic agenda. At the same time, companies like Microsoft or OpenAl are demand- 
ing the democratization of AI. What is your general opinion on this concept? 

Again, you know, the world is much more complex than a simple slogan suggests. 
Facebook is a good case in point. I am sure their initial goal was to democratize 

news. If anyone can post news, how wonderful that can be! If anyone can pro- 
vide facts on Wikipedia, how wonderful would that be! No more experts dictating 
their opinions, right? But if you really think this through - if anybody can publish 

trash about anybody and reach a worldwide audience - the benefit is not necessar- 
ily that you are making people heard that were unheard before, but you also open 

up a worldwide potential for abuse and manipulation. And that is exactly what 
we are realizing now. So democratization is fine. But the potential for massive 

manipulation and abuse is equally there in the same process, and therefore one 

has to be really careful. 


In other words: You are more or less skeptical about this concept? 

Once again, we should be careful and keep on thinking about what we are do- 
ing because tools like the Internet are so powerful. Sometimes you create things 
that have unintended consequences. And one must reevaluate the technology and 
strive to move it in a good direction. While the internet lead to democratization of 
information, we now see again massive concentration of information and power 
as well. Would you rather have a world in which only Google, Microsoft, Facebook, 
Amazon or Apple can have intelligent systems and everybody else is at their mercy 
with regard to using this technology? Would you like to have a world in which only 
one of the big tech giants can recognize anybody’s face by a machine and nobody 
else is able to? These technologies effectively encourage monopolies, that are hord- 
ing incredible amount of data and generate a lot of knowledge, but - despite the 
best intentions — at the same time also provide a lot of potential for manipulation. 
We have recently seen that this is the case with the Cambridge Analytica scandal. 
So, again, the question is: Would you like to see all data and Alto be concentrated 
with only three or four companies in the world? 

These concerns also play out on a geo-political stage. While the internet was 
designed to be a great global unifying force, it now also threatens to break into 
major regional spheres with different moral and societal attitudes that compete 
for supremacy. In China, where there are fewer laws or restrictions to data col- 
lection and handling, we see that AI feeds the emergence of an automated mass 
surveillance state that is overseen by the government. Will this - by way of compe- 
tition - undermine Western values of privacy, freedom, and independence? That 
is a 1984 Orwellian future at our doorstep, right? Indeed, democratizing it at least 
distributes the technology to a broader set of players and that is why antitrust 
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efforts are so important, domestically. But, ifwe talk about global balances - for 
example - Europe versus America: Europe does not have a large internet com- 
pany and this creates asymmetries, where one continent is critically dependent 
on Al systems from another for its information and data management. This still 
works, because relations between America and Europe are amicable and support- 
ive because both are Western democracies. But what about China, Russia, India? 
China is making dramatic progress in AI right know, and is spending billions of 
dollars on AI. So itis only a matter oftime until China and others will be on par, 
if not more advanced then the US. And, without its own clear technology base 
and vision, Europe, could be at the mercy of what other players are doing and be 
much more vulnerable to external meddling and manipulation. For me, these are 
worrisome developments. 


Nevertheless, it still makes a difference whether we talk about American or Chinese tech 
monopolies. 

Right, the so-called GAFA [Google, Apple, Facebook, Amazon] have to be consid- 
ered differently in some sense. In China, companies like Baidu, Alibaba or Ten- 
cent have very strong government connections. In the US, the playing field looks 
different because there is a public that watches these companies, and whenever 
one ofthem abuses their data power, it becomes a scandal and is immediately all 
over the news, which is very bad for the company. While I was with Facebook, I 
do have to say that - despite all the scandals - I was impressed by how much the 
company actually attempted to deal with their data in a responsible way. And the 
fact that they still produce scandals simply shows how hard it is to do that and 
how sensitive an issue it is. But I think that companies in the US also embrace 
the idea of democratizing AI because it is part of their business model. Of course, 
companies ultimately are very selfish and try to do what is best for them. But in 
the American context, protecting data is good for business, since scandals are ter- 
rible. Hence, Microsoft and Google are into democratizing AI because it supports 
their business strategies. Take Amazon, for example: One of its largest businesses 
is Amazon Web Services (AWS) that, among other things, includes renting nodes, 
so that a small company can do its computation on Amazon’s servers. Microsoft 
or Google provide similar computational resources, and if they can provide Al ser- 
vices on top of that, then it is obviously also good for their business. 
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So which would you consider to be the most important political challenges of Al in the near 
future, from your perspective as a computer scientist? 

I think that, first of all, we as scientists have a responsibility to be vigilant. But 
it gives rise to optimism that there are actually a lot of idealistic people working 
inside those big companies. Thus, the fact that there are scandals is good news, 
because it means that you cannot keep such things secret, that it forces society to 
keep thinking these things through, which is good. And regarding how politics 
should respond: Well, if you look at some of these senatorial debates, you realize 
that politicians cannot be deeply involved in every aspect of every technology, and 
hence may lack intuition about where it may go and how to respond. For this rea- 
son, I think it is very important for AI scientists to be vocal and active in a public 
dialog, so we (science, public, and governments) can ensure that we build these 
technologies to serve humanity, as opposed to serving our own political or finan- 
cial interest. 

What worries me in this context, however, is the fact that large companies are 
voraciously hiring scientists, and that universities have difficulties retaining tal- 
ented people. And we should remember that universities are (or should be) spac- 
es for open discussion and debate so that we are not manipulated by economic 
or political interests. So my point is that we should maintain a strong academic 
environment in all major areas in which Al is used. And this is a particular chal- 
lenge in Europe: Without major internet companies, it naturally suffers from a 
continuing loss of talent. With a reference to my own Al laboratory in Germany, 
I can tell you that many of the best scientists, as soon as they are done with their 
pitches or degrees, get offers that are like seven times higher (or even more) than 
the ones we can afford at a university. And when young people are being offered 
those amounts of money plus a chance at building something with a major compa- 
ny like Amazon, Apple, or Facebook, they jump at the opportunity. In other words, 
the brain drain is enormous, not just from academia to industry, but between 
countries. Therefore, in an age of AI, Europe must move much more aggressively 
to provide for its future. 


What can Europe do to change that? And how do politicians, people, or the public know 
which experts they can trust? 

Well, as to your last question, I think by being less risk averse and doing more 
to encourage technology disruption: Europe has outstanding scientists and en- 
gineers. There is also outstanding support and freedom in Europe to carry out 
innovative and fundamental science. Europe has very bright, well-educated, and 
idealistic scientists. I could even argue that many of the top scientists in America 
were trained or started their career in Europe. That is not just an empty phrase, I 
could name many famous examples. But the area in which we are doing badly is 
letting the scientific advances challenge the status-quo in society. What is needed 
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is fast, practical, and disruptive moonshot projects. DARPA in the US, for exam- 
ple, has done that very successfully for the government. And so do companies like 
Google, Tesla, Amazon, all of which did not exist 30 years ago. 

The other thing that should be improved is the technology transfer into in- 
dustrial exploitation. In Europe, we actually have many entrepreneurs who start 
companies. The risk takers are there, the young people are there, the bright ideas 
are there, and the excitement and the eagerness to do this are there. What’s miss- 
ing is capital to support such activities and also more willingness and speed of 
mergers and acquisitions. For example, in the US, small companies are bought 
up very quickly. Some of the companies exist only twelve or eighteen months be- 
fore they are being absorbed by a larger corporation. This is a healthy process as it 
speeds the transition from idea to concept to product to industry. But in Europe, 
that is very rare. Here, it is very difficult for companies to be bought. It takes a 
long time to go public, to enter the stock market and so on. The transition from a 
small successful start-up to a large business has so much friction in Europe that it 
misses many opportunities; speed is of the essence in this kind of game. And this 
ultimately drives many small companies and their young entrepreneurs to the US 
and China. 


Another political-ethical concern that many people talk about these days is the problem of 
algorithmic biases. How are these problems related to your research in natural language 
processing and the translation of spoken languages? 
I am glad that you are bringing us back to this topic. So far, we talked a lot about 
how AI affects society. Another important political dimension is to discuss how 
we pick projects that contribute to a society that we want to live in. And for me that 
means speech translation, because I think this is one of the big problems in Eu- 
rope. Europeans speak 23 different languages, and these are only the official ones. 
In fact, there are many more languages in Europe. And this situation generates 
separation, misunderstandings, and also a considerable loss in business opportu- 
nities. One big reason why e-commerce is more challenging in Europe is because 
it is so fragmented. Each country in Europe has a different legal system, a differ- 
ent delivery system, and much of that is of course fossilized in language, because 
if everything has to be done in multiple languages, it complicates transnational 
business exchanges. But saying that everyone should learn English, Esperanto, or 
something like that would be ridiculous and also not desirable. In fact, having the 
variety and diversity of languages is something Europeans are rightfully proud of. 
Against this background, technology must not be regarded as an obstacle, but 
as a tremendous problem solver if we want to develop a technology capable of text 
and speech translation that bridges these language barriers on all fronts, so that 
we actually have a language-transparent world. Imagine you are going to China, 
Russia, or Spain, and like to operate in these countries as if you are at home, with- 


“That is a 1984 Orwellian future at our doorstep, right?” 323 


out any language disadvantages. But if you think that through, what such a sce- 
nario would mean if you are in all these countries without knowing the respective 
language, what all kind of assistance it would require so that you do not notice 
the language barrier anymore. And indeed to achieve this is the very vision we are 


working on. 
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